2024-09-10T19:46:50.839 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T19:46:50.853 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:46:50.942 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899522 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: skanta@redhat.com first_in_suite: false job_id: '7899522' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBInNopY9cL86J8Dr/eE0q+sLH3vmS/qmjPc5QSa2UNHuhgb1NXV/aMkP3/HvnJQMiBARxweotMeIVsS/7IN3/PY= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T19:46:50.942 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T19:46:50.943 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T19:46:50.943 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T19:46:50.944 INFO:teuthology.task.internal:Checking packages... 2024-09-10T19:46:50.967 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T19:46:50.968 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T19:46:50.968 INFO:teuthology.packaging:ref: None 2024-09-10T19:46:50.968 INFO:teuthology.packaging:tag: None 2024-09-10T19:46:50.968 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:46:50.968 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:46:50.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:46:51.108 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858-gbad1f558-1jammy 2024-09-10T19:46:51.109 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T19:46:51.116 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T19:46:51.117 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T19:46:51.122 INFO:teuthology.task.internal:Saving configuration 2024-09-10T19:46:51.133 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T19:46:51.146 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T19:46:51.172 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899522', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 19:40:26.896386', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOrKOBTTLvDDkBO69+pqUksMnGWxggv51gMXD3jbbSFY'} 2024-09-10T19:46:51.173 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T19:46:51.214 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-10T19:46:51.214 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T19:46:51.305 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb3b97ff40>, signals=[15]) 2024-09-10T19:46:51.305 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T19:46:51.311 INFO:teuthology.task.internal:Opening connections... 2024-09-10T19:46:51.311 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-10T19:46:51.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:46:51.397 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T19:46:51.403 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-10T19:46:51.502 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-10T19:46:51.502 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-10T19:46:51.547 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-10T19:46:51.547 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Ubuntu" 2024-09-10T19:46:51.547 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="22.04" 2024-09-10T19:46:51.547 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-10T19:46:51.547 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_CODENAME=jammy 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:ID=ubuntu 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE=debian 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-10T19:46:51.548 INFO:teuthology.orchestra.run.smithi039.stdout:UBUNTU_CODENAME=jammy 2024-09-10T19:46:51.549 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-10T19:46:51.621 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T19:46:51.630 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T19:46:51.636 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T19:46:51.637 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-10T19:46:51.641 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T19:46:51.647 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T19:46:51.647 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-10T19:46:51.689 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T19:46:51.719 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T19:46:51.732 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T19:46:51.732 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T19:46:51.733 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T19:46:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:46:51.738 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-10T19:46:51.783 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-56-generic 2024-09-10T19:46:51.783 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.15.0-56-generic 2024-09-10T19:46:51.783 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-10T19:46:51.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-10T19:46:52.064 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T19:46:52.093 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-10T19:46:52.158 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-10T19:46:52.158 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-10T19:46:52.295 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-10T19:46:52.357 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,987 kB] 2024-09-10T19:46:52.390 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-10T19:46:52.394 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-10T19:46:52.394 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-10T19:46:52.395 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-10T19:46:52.416 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-10T19:46:52.419 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-10T19:46:52.419 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-10T19:46:52.426 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,122 kB] 2024-09-10T19:46:52.434 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-10T19:46:52.436 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-10T19:46:52.482 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-10T19:46:52.482 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-10T19:46:52.483 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-10T19:46:52.483 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-10T19:46:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-10T19:46:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-10T19:46:52.485 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-10T19:46:52.485 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-10T19:46:52.500 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-10T19:46:52.517 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-10T19:46:52.517 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-10T19:46:52.517 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-10T19:46:52.639 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-10T19:46:53.057 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-10T19:46:53.073 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-10T19:46:53.082 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-10T19:46:53.082 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-10T19:46:53.083 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-10T19:46:53.179 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-10T19:46:53.190 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-10T19:46:53.190 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-10T19:46:53.213 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-10T19:46:53.232 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-10T19:46:53.237 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-10T19:46:53.237 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-10T19:46:53.238 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-10T19:46:53.238 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-10T19:46:53.317 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-10T19:46:59.061 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 15.8 MB in 2s (7,148 kB/s) 2024-09-10T19:47:00.408 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-10T19:47:00.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T19:47:00.493 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-10T19:47:00.703 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-10T19:47:00.704 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-10T19:47:00.902 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T19:47:00.902 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T19:47:00.902 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T19:47:00.903 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T19:47:00.903 INFO:teuthology.orchestra.run.smithi039.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-10T19:47:00.903 INFO:teuthology.orchestra.run.smithi039.stdout: python2.7-minimal 2024-09-10T19:47:00.903 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T19:47:00.904 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-10T19:47:00.904 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-10T19:47:00.904 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-10T19:47:00.905 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-10T19:47:00.906 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-10T19:47:00.906 INFO:teuthology.orchestra.run.smithi039.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-10T19:47:00.906 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-10T19:47:00.906 INFO:teuthology.orchestra.run.smithi039.stdout: thermald 2024-09-10T19:47:00.957 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-10T19:47:00.957 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-10T19:47:00.957 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-10T19:47:00.957 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-10T19:47:00.958 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages will be upgraded: 2024-09-10T19:47:00.959 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-10T19:47:01.140 INFO:teuthology.orchestra.run.smithi039.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-10T19:47:01.140 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 113 MB of archives. 2024-09-10T19:47:01.140 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-10T19:47:01.140 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-10T19:47:02.569 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-10T19:47:02.942 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-10T19:47:05.177 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-10T19:47:05.177 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-10T19:47:05.178 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-10T19:47:05.585 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-10T19:47:05.644 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-10T19:47:06.075 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 113 MB in 5s (24.2 MB/s) 2024-09-10T19:47:06.249 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-10T19:47:08.180 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-10T19:47:08.185 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T19:47:08.246 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:10.930 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-10T19:47:10.952 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T19:47:11.051 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:11.449 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-10T19:47:11.470 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T19:47:11.499 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:17.705 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T19:47:17.832 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T19:47:18.165 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T19:47:18.302 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T19:47:18.571 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-10T19:47:18.591 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-10T19:47:18.630 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-10T19:47:24.060 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-10T19:47:24.097 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T19:47:24.134 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:27.423 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T19:47:28.448 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T19:47:28.774 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-10T19:47:29.012 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:29.138 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-10T19:47:29.267 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:30.620 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-10T19:47:30.620 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-10T19:47:30.621 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-10T19:47:30.621 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-10T19:47:30.777 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:32.212 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:34.389 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-10T19:47:34.598 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-10T19:47:34.725 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T19:47:34.793 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-10T19:47:34.793 INFO:teuthology.orchestra.run.smithi039.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-10T19:47:49.197 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-10T19:47:49.197 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub' 2024-09-10T19:47:49.208 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T19:47:49.234 INFO:teuthology.orchestra.run.smithi039.stdout:Generating grub configuration file ... 2024-09-10T19:47:49.500 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-10T19:47:49.527 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-10T19:47:49.798 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-10T19:47:49.804 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-10T19:47:49.877 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-10T19:47:49.883 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-10T19:47:50.046 INFO:teuthology.orchestra.run.smithi039.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-10T19:47:50.047 INFO:teuthology.orchestra.run.smithi039.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-10T19:47:50.047 INFO:teuthology.orchestra.run.smithi039.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-10T19:47:50.085 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-09-10T19:47:50.549 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-10T19:47:50.574 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-10T19:47:50.574 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-10T19:47:50.574 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-10T19:47:50.574 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.119.119 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-10T19:47:50.575 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-10T19:47:50.575 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-10T19:47:50.576 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T19:47:50.576 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T19:47:50.576 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T19:47:50.576 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T19:47:50.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-10T19:47:50.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-10T19:47:50.880 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T19:47:50.889 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-10T19:47:50.907 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-10T19:47:51.002 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-10T19:47:52.357 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-10T19:47:52.376 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T19:47:52.432 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-10T19:47:52.622 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-10T19:47:52.623 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-10T19:47:52.803 INFO:teuthology.orchestra.run.smithi039.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-10T19:47:52.803 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T19:47:52.803 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T19:47:52.803 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T19:47:52.803 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T19:47:52.804 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-10T19:47:52.804 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-10T19:47:52.804 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T19:47:52.845 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-10T19:47:52.846 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-10T19:47:52.907 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.119.119 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T19:47:52.908 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-10T19:47:52.909 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-10T19:47:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-10T19:47:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-10T19:47:52.909 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-10T19:47:52.954 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.NMgdSZ5r8F 2024-09-10T19:47:52.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NMgdSZ5r8F 2024-09-10T19:47:53.011 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.NMgdSZ5r8F 2024-09-10T19:47:53.132 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.NMgdSZ5r8F is 10KB 2024-09-10T19:47:53.146 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.NMgdSZ5r8F 2024-09-10T19:47:53.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-10T19:47:53.207 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T19:47:53.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-10T19:47:53.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-10T19:47:53.275 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-10T19:47:53.275 DEBUG:teuthology.orchestra.run.smithi039:> sudo update-grub 2024-09-10T19:47:54.455 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub' 2024-09-10T19:47:54.456 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T19:47:54.458 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-10T19:47:54.659 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-10T19:47:54.666 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-10T19:47:54.921 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-10T19:47:54.926 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-10T19:47:54.969 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-10T19:47:54.974 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-10T19:47:55.103 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-10T19:47:55.104 INFO:teuthology.orchestra.run.smithi039.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-10T19:47:55.104 INFO:teuthology.orchestra.run.smithi039.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-10T19:47:55.124 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-10T19:47:55.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-10T19:48:25.157 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T19:48:25.158 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:48:25.159 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:48:43.709 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-10T19:48:52.717 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:48:52.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:48:55.773 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-10T19:49:07.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:49:07.785 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:49:10.845 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-10T19:49:25.857 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:49:25.858 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:50:25.920 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T19:50:43.937 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:50:43.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:50:44.164 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-10T19:50:44.838 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:50:44.839 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-10T19:50:44.839 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-10T19:50:44.886 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-119-generic 2024-09-10T19:50:44.886 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-10T19:50:44.886 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T19:50:44.886 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-10T19:50:45.888 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T19:50:45.888 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T19:50:45.966 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-10T19:50:46.007 DEBUG:teuthology.parallel:result is None 2024-09-10T19:50:46.007 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T19:50:46.014 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T19:50:46.015 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T19:50:46.020 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T19:50:46.086 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T19:50:46.099 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T19:50:46.099 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T19:50:46.126 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T19:50:46.171 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T19:50:46.172 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T19:50:46.196 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:50:46.206 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:50:46.208 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T19:50:46.248 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T19:50:46.248 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T19:50:46.261 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T19:50:46.377 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T19:50:46.377 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T19:50:46.383 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T19:50:46.430 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T19:50:46.478 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T19:50:46.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T19:50:46.546 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-10T19:50:46.622 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T19:50:46.630 INFO:teuthology.task.internal:Starting timer... 2024-09-10T19:50:46.630 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T19:50:46.681 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T19:50:46.735 DEBUG:teuthology.task.selinux:Excluding smithi039: OS 'ubuntu' does not support SELinux 2024-09-10T19:50:46.735 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T19:50:46.736 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-10T19:50:46.736 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T19:50:46.736 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T19:50:46.817 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T19:50:46.830 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T19:50:46.831 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T19:58:24.646 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com')] 2024-09-10T19:58:24.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:58:24.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:58:24.736 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-10T19:58:24.813 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-10T19:58:24.814 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T19:58:24.824 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T19:58:24.824 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T19:58:24.825 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Command line: ntpd -gq 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: ---------------------------------------------------- 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: ntp-4 is maintained by Network Time Foundation, 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: corporation. Support and training for ntp-4 are 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: available at https://www.nwtime.org/support 2024-09-10T19:58:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: ---------------------------------------------------- 2024-09-10T19:58:24.886 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: proto: precision = 0.061 usec (-24) 2024-09-10T19:58:24.886 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: basedate set to 2022-02-04 2024-09-10T19:58:24.886 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: gps base set to 2022-02-06 (week 2196) 2024-09-10T19:58:24.886 INFO:teuthology.orchestra.run.smithi039.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T19:58:24.887 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T19:58:24.887 INFO:teuthology.orchestra.run.smithi039.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: restrict ::: KOD does nothing without LIMITED. 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listen and drop on 0 v6wildcard [::]:123 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listen normally on 2 lo 127.0.0.1:123 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listen normally on 3 ens1f0 172.21.15.39:123 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listen normally on 4 lo [::1]:123 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1370%4]:123 2024-09-10T19:58:24.888 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:24 ntpd[18356]: Listening on routing socket on fd #22 for interface updates 2024-09-10T19:58:25.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:25 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:25.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:25 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:26.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:26.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:26 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:26.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:26.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:26 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:26.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:26 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:26.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:26 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:27.889 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:27.889 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:27 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:27.889 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:27.889 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:27 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:27.889 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:27 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:27.889 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:27 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:28.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:28.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:28 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:28.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:28.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:28 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:28.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:28 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:28.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:28 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:29.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:29.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:29.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:29.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:29.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:29 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:30.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:30 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:30 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:30 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:30 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:30 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:30 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:31.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:31.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:31.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:31.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:31.887 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:31 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:32.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:32.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:32.885 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stdout:10 Sep 19:58:32 ntpd[18356]: ntpd: time slew +0.001591 s 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stdout:ntpd: time slew +0.001591s 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-10T19:58:32.886 INFO:teuthology.orchestra.run.smithi039.stderr:10 Sep 19:58:32 ntpd[18356]: can't open /var/log/ntpstats/loopstats.20240910: Permission denied 2024-09-10T19:58:32.937 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T19:58:32.937 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-10T19:58:32.937 INFO:teuthology.orchestra.run.smithi039.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T19:58:32.937 INFO:teuthology.orchestra.run.smithi039.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T19:58:32.937 INFO:teuthology.orchestra.run.smithi039.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T19:58:32.937 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T19:58:32.939 INFO:teuthology.run_tasks:Running task install... 2024-09-10T19:58:32.950 DEBUG:teuthology.task.install:project ceph 2024-09-10T19:58:32.950 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T19:58:32.950 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T19:58:32.950 INFO:teuthology.task.install:Using flavor: default 2024-09-10T19:58:32.955 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T19:58:32.955 INFO:teuthology.task.install:extra packages: [] 2024-09-10T19:58:32.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-key list | grep Ceph 2024-09-10T19:58:33.038 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-10T19:58:33.069 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-10T19:58:33.069 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph.com (release key) 2024-09-10T19:58:33.069 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-10T19:58:33.070 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:58:33.213 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default/ 2024-09-10T19:58:33.213 INFO:teuthology.task.install.deb:Package version is 19.3.0-4858-gbad1f558-1jammy 2024-09-10T19:58:33.326 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T19:58:33.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-10T19:58:33.341 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-10T19:58:33.518 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-10T19:58:33.572 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy InRelease 2024-09-10T19:58:33.593 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-10T19:58:33.615 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-10T19:58:33.636 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-10T19:58:33.642 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T19:58:33.663 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2024-09-10T19:58:33.720 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-10T19:58:33.802 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-10T19:58:33.958 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 29.3 kB in 0s (69.2 kB/s) 2024-09-10T19:58:35.001 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-10T19:58:35.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4858-gbad1f558-1jammy cephadm=19.3.0-4858-gbad1f558-1jammy ceph-mds=19.3.0-4858-gbad1f558-1jammy ceph-mgr=19.3.0-4858-gbad1f558-1jammy ceph-common=19.3.0-4858-gbad1f558-1jammy ceph-fuse=19.3.0-4858-gbad1f558-1jammy ceph-test=19.3.0-4858-gbad1f558-1jammy ceph-volume=19.3.0-4858-gbad1f558-1jammy radosgw=19.3.0-4858-gbad1f558-1jammy python3-rados=19.3.0-4858-gbad1f558-1jammy python3-rgw=19.3.0-4858-gbad1f558-1jammy python3-cephfs=19.3.0-4858-gbad1f558-1jammy python3-rbd=19.3.0-4858-gbad1f558-1jammy libcephfs2=19.3.0-4858-gbad1f558-1jammy libcephfs-dev=19.3.0-4858-gbad1f558-1jammy librados2=19.3.0-4858-gbad1f558-1jammy librbd1=19.3.0-4858-gbad1f558-1jammy rbd-fuse=19.3.0-4858-gbad1f558-1jammy 2024-09-10T19:58:35.084 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-10T19:58:35.298 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-10T19:58:35.298 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-10T19:58:35.518 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T19:58:35.518 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T19:58:35.518 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T19:58:35.518 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T19:58:35.519 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-10T19:58:35.519 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-10T19:58:35.519 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T19:58:35.520 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-10T19:58:35.520 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-10T19:58:35.521 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-10T19:58:35.521 INFO:teuthology.orchestra.run.smithi039.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-10T19:58:35.521 INFO:teuthology.orchestra.run.smithi039.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-10T19:58:35.521 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-10T19:58:35.523 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-10T19:58:35.524 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet zip 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-10T19:58:35.525 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-10T19:58:35.526 INFO:teuthology.orchestra.run.smithi039.stdout: btrfs-tools 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-10T19:58:35.633 INFO:teuthology.orchestra.run.smithi039.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-10T19:58:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-10T19:58:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-10T19:58:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-10T19:58:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-10T19:58:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-10T19:58:35.635 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-10T19:58:35.636 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-10T19:58:35.637 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-10T19:58:35.637 INFO:teuthology.orchestra.run.smithi039.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-10T19:58:35.780 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-10T19:58:35.780 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 251 MB of archives. 2024-09-10T19:58:35.781 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 932 MB of additional disk space will be used. 2024-09-10T19:58:35.781 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4858-gbad1f558-1jammy [3,634 kB] 2024-09-10T19:58:35.825 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-10T19:58:35.968 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4858-gbad1f558-1jammy [3,330 kB] 2024-09-10T19:58:36.015 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4858-gbad1f558-1jammy [722 kB] 2024-09-10T19:58:36.026 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4858-gbad1f558-1jammy [357 kB] 2024-09-10T19:58:36.032 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4858-gbad1f558-1jammy [32.8 kB] 2024-09-10T19:58:36.032 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4858-gbad1f558-1jammy [180 kB] 2024-09-10T19:58:36.035 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4858-gbad1f558-1jammy [71.2 kB] 2024-09-10T19:58:36.036 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4858-gbad1f558-1jammy [338 kB] 2024-09-10T19:58:36.042 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4858-gbad1f558-1jammy [7,987 kB] 2024-09-10T19:58:36.174 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4858-gbad1f558-1jammy [112 kB] 2024-09-10T19:58:36.176 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4858-gbad1f558-1jammy [473 kB] 2024-09-10T19:58:36.182 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4858-gbad1f558-1jammy [28.3 MB] 2024-09-10T19:58:36.249 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-10T19:58:36.261 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-10T19:58:36.342 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-10T19:58:36.572 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-10T19:58:36.583 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-10T19:58:36.613 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-10T19:58:36.621 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-10T19:58:36.623 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-10T19:58:36.624 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-10T19:58:36.625 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-10T19:58:36.643 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-10T19:58:36.647 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-10T19:58:36.709 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-10T19:58:36.775 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-10T19:58:36.776 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4858-gbad1f558-1jammy [5,206 kB] 2024-09-10T19:58:36.776 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-10T19:58:36.780 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-10T19:58:36.784 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-10T19:58:36.787 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-10T19:58:36.789 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-10T19:58:36.789 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-10T19:58:36.792 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-10T19:58:36.836 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4858-gbad1f558-1jammy [251 kB] 2024-09-10T19:58:36.838 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4858-gbad1f558-1jammy [124 kB] 2024-09-10T19:58:36.840 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4858-gbad1f558-1jammy [1,560 kB] 2024-09-10T19:58:36.853 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4858-gbad1f558-1jammy [6,309 kB] 2024-09-10T19:58:36.861 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-10T19:58:36.874 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-10T19:58:36.875 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-10T19:58:36.875 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-10T19:58:36.875 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-10T19:58:36.875 INFO:teuthology.orchestra.run.smithi039.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-10T19:58:36.878 INFO:teuthology.orchestra.run.smithi039.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-10T19:58:36.943 INFO:teuthology.orchestra.run.smithi039.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-10T19:58:36.943 INFO:teuthology.orchestra.run.smithi039.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-10T19:58:36.945 INFO:teuthology.orchestra.run.smithi039.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-10T19:58:36.953 INFO:teuthology.orchestra.run.smithi039.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4858-gbad1f558-1jammy [23.1 MB] 2024-09-10T19:58:37.024 INFO:teuthology.orchestra.run.smithi039.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-10T19:58:37.024 INFO:teuthology.orchestra.run.smithi039.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-10T19:58:37.025 INFO:teuthology.orchestra.run.smithi039.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-10T19:58:37.027 INFO:teuthology.orchestra.run.smithi039.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-10T19:58:37.029 INFO:teuthology.orchestra.run.smithi039.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-10T19:58:37.030 INFO:teuthology.orchestra.run.smithi039.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-10T19:58:37.105 INFO:teuthology.orchestra.run.smithi039.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-10T19:58:37.106 INFO:teuthology.orchestra.run.smithi039.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-10T19:58:37.109 INFO:teuthology.orchestra.run.smithi039.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-10T19:58:37.109 INFO:teuthology.orchestra.run.smithi039.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-10T19:58:37.187 INFO:teuthology.orchestra.run.smithi039.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-10T19:58:37.191 INFO:teuthology.orchestra.run.smithi039.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-10T19:58:37.192 INFO:teuthology.orchestra.run.smithi039.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-10T19:58:37.195 INFO:teuthology.orchestra.run.smithi039.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-10T19:58:37.196 INFO:teuthology.orchestra.run.smithi039.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-10T19:58:37.201 INFO:teuthology.orchestra.run.smithi039.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-10T19:58:37.268 INFO:teuthology.orchestra.run.smithi039.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-10T19:58:37.270 INFO:teuthology.orchestra.run.smithi039.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-10T19:58:37.274 INFO:teuthology.orchestra.run.smithi039.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-10T19:58:37.402 INFO:teuthology.orchestra.run.smithi039.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4858-gbad1f558-1jammy [14.0 kB] 2024-09-10T19:58:37.402 INFO:teuthology.orchestra.run.smithi039.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4858-gbad1f558-1jammy [880 kB] 2024-09-10T19:58:37.410 INFO:teuthology.orchestra.run.smithi039.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4858-gbad1f558-1jammy [2,453 kB] 2024-09-10T19:58:37.433 INFO:teuthology.orchestra.run.smithi039.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4858-gbad1f558-1jammy [813 kB] 2024-09-10T19:58:37.435 INFO:teuthology.orchestra.run.smithi039.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-10T19:58:37.437 INFO:teuthology.orchestra.run.smithi039.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-10T19:58:37.478 INFO:teuthology.orchestra.run.smithi039.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4858-gbad1f558-1jammy [160 kB] 2024-09-10T19:58:37.478 INFO:teuthology.orchestra.run.smithi039.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-10T19:58:37.478 INFO:teuthology.orchestra.run.smithi039.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-10T19:58:37.480 INFO:teuthology.orchestra.run.smithi039.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4858-gbad1f558-1jammy [61.4 MB] 2024-09-10T19:58:37.480 INFO:teuthology.orchestra.run.smithi039.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-10T19:58:37.512 INFO:teuthology.orchestra.run.smithi039.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-10T19:58:37.514 INFO:teuthology.orchestra.run.smithi039.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-10T19:58:37.515 INFO:teuthology.orchestra.run.smithi039.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-10T19:58:37.596 INFO:teuthology.orchestra.run.smithi039.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-10T19:58:37.596 INFO:teuthology.orchestra.run.smithi039.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-10T19:58:37.602 INFO:teuthology.orchestra.run.smithi039.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-10T19:58:37.602 INFO:teuthology.orchestra.run.smithi039.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-10T19:58:37.603 INFO:teuthology.orchestra.run.smithi039.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-10T19:58:37.604 INFO:teuthology.orchestra.run.smithi039.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-10T19:58:37.604 INFO:teuthology.orchestra.run.smithi039.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-10T19:58:37.605 INFO:teuthology.orchestra.run.smithi039.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-10T19:58:37.610 INFO:teuthology.orchestra.run.smithi039.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-10T19:58:37.684 INFO:teuthology.orchestra.run.smithi039.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-10T19:58:37.686 INFO:teuthology.orchestra.run.smithi039.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-10T19:58:37.687 INFO:teuthology.orchestra.run.smithi039.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-10T19:58:37.765 INFO:teuthology.orchestra.run.smithi039.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-10T19:58:37.771 INFO:teuthology.orchestra.run.smithi039.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-10T19:58:37.774 INFO:teuthology.orchestra.run.smithi039.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-10T19:58:37.775 INFO:teuthology.orchestra.run.smithi039.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-10T19:58:37.787 INFO:teuthology.orchestra.run.smithi039.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-10T19:58:37.788 INFO:teuthology.orchestra.run.smithi039.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-10T19:58:37.796 INFO:teuthology.orchestra.run.smithi039.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-10T19:58:37.847 INFO:teuthology.orchestra.run.smithi039.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-10T19:58:37.848 INFO:teuthology.orchestra.run.smithi039.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-10T19:58:37.849 INFO:teuthology.orchestra.run.smithi039.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-10T19:58:37.928 INFO:teuthology.orchestra.run.smithi039.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-10T19:58:37.930 INFO:teuthology.orchestra.run.smithi039.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-10T19:58:37.948 INFO:teuthology.orchestra.run.smithi039.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-10T19:58:37.949 INFO:teuthology.orchestra.run.smithi039.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-10T19:58:37.950 INFO:teuthology.orchestra.run.smithi039.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-10T19:58:37.955 INFO:teuthology.orchestra.run.smithi039.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-10T19:58:37.956 INFO:teuthology.orchestra.run.smithi039.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-10T19:58:38.700 INFO:teuthology.orchestra.run.smithi039.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4858-gbad1f558-1jammy [8,625 kB] 2024-09-10T19:58:38.812 INFO:teuthology.orchestra.run.smithi039.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4858-gbad1f558-1jammy [14.2 kB] 2024-09-10T19:58:38.812 INFO:teuthology.orchestra.run.smithi039.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4858-gbad1f558-1jammy [55.3 MB] 2024-09-10T19:58:39.968 INFO:teuthology.orchestra.run.smithi039.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4858-gbad1f558-1jammy [130 kB] 2024-09-10T19:58:39.970 INFO:teuthology.orchestra.run.smithi039.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4858-gbad1f558-1jammy [39.0 kB] 2024-09-10T19:58:39.971 INFO:teuthology.orchestra.run.smithi039.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4858-gbad1f558-1jammy [15.9 MB] 2024-09-10T19:58:40.268 INFO:teuthology.orchestra.run.smithi039.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4858-gbad1f558-1jammy [91.2 kB] 2024-09-10T19:58:40.876 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 251 MB in 5s (54.2 MB/s) 2024-09-10T19:58:40.994 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-10T19:58:41.053 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-10T19:58:41.058 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-10T19:58:41.143 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-10T19:58:41.413 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-10T19:58:41.440 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-10T19:58:41.488 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-10T19:58:41.816 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-10T19:58:41.843 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-10T19:58:41.883 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-10T19:58:42.320 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-10T19:58:42.346 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T19:58:42.394 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T19:58:42.765 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-10T19:58:42.791 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T19:58:42.831 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T19:58:43.151 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-10T19:58:43.177 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T19:58:43.217 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T19:58:43.570 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-10T19:58:43.597 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-10T19:58:43.637 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-10T19:58:43.973 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librados2. 2024-09-10T19:58:44.000 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../007-librados2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:44.040 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librados2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:44.468 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libnbd0. 2024-09-10T19:58:44.494 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-10T19:58:44.526 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-10T19:58:44.837 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librbd1. 2024-09-10T19:58:44.864 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../009-librbd1_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:44.903 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librbd1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:45.407 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs2. 2024-09-10T19:58:45.434 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:45.473 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:45.750 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rados. 2024-09-10T19:58:45.766 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:45.800 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rados (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:46.053 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-10T19:58:46.079 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:58:46.119 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-argparse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:46.372 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cephfs. 2024-09-10T19:58:46.388 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:46.430 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cephfs (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:46.741 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-10T19:58:46.768 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:58:46.808 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:47.136 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-10T19:58:47.163 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-10T19:58:47.202 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-10T19:58:47.472 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-prettytable. 2024-09-10T19:58:47.499 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-10T19:58:47.530 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-10T19:58:47.783 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rbd. 2024-09-10T19:58:47.799 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:47.841 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rbd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:48.478 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-10T19:58:48.504 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-10T19:58:48.544 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-10T19:58:48.830 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-10T19:58:48.856 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-10T19:58:48.896 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-10T19:58:49.158 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-10T19:58:49.173 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-10T19:58:49.207 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-10T19:58:49.561 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua5.1. 2024-09-10T19:58:49.587 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-10T19:58:49.627 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-10T19:58:49.939 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-any. 2024-09-10T19:58:49.965 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-10T19:58:50.005 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-10T19:58:50.225 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package zip. 2024-09-10T19:58:50.240 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-10T19:58:50.274 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking zip (3.0-12build2) ... 2024-09-10T19:58:50.561 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package unzip. 2024-09-10T19:58:50.587 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-10T19:58:50.636 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-10T19:58:51.015 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package luarocks. 2024-09-10T19:58:51.042 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-10T19:58:51.081 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-10T19:58:51.439 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librgw2. 2024-09-10T19:58:51.466 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../026-librgw2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:51.497 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librgw2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:52.083 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rgw. 2024-09-10T19:58:52.099 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:52.141 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:52.477 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-10T19:58:52.504 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-10T19:58:52.544 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-10T19:58:52.838 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libradosstriper1. 2024-09-10T19:58:52.865 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:52.905 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libradosstriper1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:53.258 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-common. 2024-09-10T19:58:53.285 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:53.316 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:54.855 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-base. 2024-09-10T19:58:54.882 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:58:54.931 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-base (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:58:55.551 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-10T19:58:55.578 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-10T19:58:55.617 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-10T19:58:55.912 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cheroot. 2024-09-10T19:58:55.938 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-10T19:58:55.978 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-10T19:58:56.281 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-10T19:58:56.308 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-10T19:58:56.356 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-10T19:58:56.642 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-10T19:58:56.669 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-10T19:58:56.709 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-10T19:58:56.928 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-10T19:58:56.955 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-10T19:58:56.995 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-10T19:58:57.281 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempora. 2024-09-10T19:58:57.308 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-10T19:58:57.347 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-10T19:58:57.667 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-portend. 2024-09-10T19:58:57.694 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-10T19:58:57.733 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-10T19:58:57.995 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-10T19:58:58.022 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-10T19:58:58.053 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-10T19:58:58.331 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-10T19:58:58.358 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-10T19:58:58.397 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-10T19:58:58.695 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-natsort. 2024-09-10T19:58:58.722 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-10T19:58:58.753 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-10T19:58:59.056 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-logutils. 2024-09-10T19:58:59.083 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-10T19:58:59.122 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-10T19:58:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-10T19:58:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-10T19:58:59.476 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-10T19:58:59.754 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-mako. 2024-09-10T19:58:59.769 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-10T19:58:59.803 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-10T19:59:00.082 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-10T19:59:00.098 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-10T19:59:00.132 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-10T19:59:00.393 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-10T19:59:00.420 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-10T19:59:00.460 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-10T19:59:00.763 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webob. 2024-09-10T19:59:00.790 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-10T19:59:00.829 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-10T19:59:01.174 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-waitress. 2024-09-10T19:59:01.201 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-10T19:59:01.244 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-10T19:59:01.519 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempita. 2024-09-10T19:59:01.546 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-10T19:59:01.585 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-10T19:59:01.855 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-paste. 2024-09-10T19:59:01.882 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-10T19:59:01.921 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-10T19:59:02.233 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-10T19:59:02.260 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-10T19:59:02.300 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-10T19:59:02.570 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-10T19:59:02.597 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-10T19:59:02.636 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-10T19:59:02.939 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webtest. 2024-09-10T19:59:02.966 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-10T19:59:03.006 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-10T19:59:03.317 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pecan. 2024-09-10T19:59:03.344 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-10T19:59:03.392 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-10T19:59:03.956 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-certifi. 2024-09-10T19:59:03.983 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-10T19:59:04.022 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-10T19:59:04.300 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-idna. 2024-09-10T19:59:04.327 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-10T19:59:04.358 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-10T19:59:04.662 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-urllib3. 2024-09-10T19:59:04.689 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-10T19:59:04.728 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-10T19:59:05.048 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests. 2024-09-10T19:59:05.075 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-10T19:59:05.114 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-10T19:59:05.409 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-10T19:59:05.436 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-10T19:59:05.484 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-10T19:59:05.796 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-10T19:59:05.812 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:59:05.854 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:06.209 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-10T19:59:06.237 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:06.267 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:06.570 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-10T19:59:06.597 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-10T19:59:06.637 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-10T19:59:06.940 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-openssl. 2024-09-10T19:59:06.967 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-10T19:59:07.006 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-10T19:59:07.318 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr. 2024-09-10T19:59:07.345 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:07.384 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:07.738 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mon. 2024-09-10T19:59:07.765 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:07.804 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mon (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:08.400 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-osd. 2024-09-10T19:59:08.427 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:08.466 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-osd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:10.103 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph. 2024-09-10T19:59:10.119 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../067-ceph_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:10.153 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:10.447 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-fuse. 2024-09-10T19:59:10.478 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:10.514 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:10.875 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mds. 2024-09-10T19:59:10.902 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:10.942 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mds (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:11.337 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package cephadm. 2024-09-10T19:59:11.364 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../070-cephadm_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:11.403 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-10T19:59:11.691 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-10T19:59:11.731 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-10T19:59:12.033 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-10T19:59:12.050 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-10T19:59:12.083 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-10T19:59:13.089 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-babel. 2024-09-10T19:59:13.116 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-10T19:59:13.155 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-10T19:59:13.483 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jinja2. 2024-09-10T19:59:13.511 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-10T19:59:13.550 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-10T19:59:13.878 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-10T19:59:13.905 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:59:13.945 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:14.507 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-10T19:59:14.534 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-10T19:59:14.566 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-10T19:59:14.834 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-routes. 2024-09-10T19:59:14.862 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-10T19:59:14.909 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-10T19:59:15.196 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-10T19:59:15.223 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:59:15.262 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:17.859 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-10T19:59:17.877 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-10T19:59:17.943 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-10T19:59:18.405 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-joblib. 2024-09-10T19:59:18.433 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-10T19:59:18.463 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-10T19:59:18.809 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-10T19:59:18.837 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-10T19:59:18.875 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-10T19:59:19.186 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn. 2024-09-10T19:59:19.214 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-10T19:59:19.338 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-10T19:59:20.040 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-10T19:59:20.069 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:59:20.106 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:21.068 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cachetools. 2024-09-10T19:59:21.096 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-10T19:59:21.134 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-10T19:59:21.379 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-10T19:59:21.407 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-10T19:59:21.446 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-10T19:59:21.766 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-10T19:59:21.783 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-10T19:59:21.824 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-10T19:59:22.153 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rsa. 2024-09-10T19:59:22.181 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-10T19:59:22.219 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-10T19:59:22.531 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-google-auth. 2024-09-10T19:59:22.559 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-10T19:59:22.598 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-10T19:59:22.901 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-10T19:59:22.918 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-10T19:59:22.951 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-10T19:59:23.263 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-websocket. 2024-09-10T19:59:23.291 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-10T19:59:23.329 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-10T19:59:23.624 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-10T19:59:23.653 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-10T19:59:23.690 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-10T19:59:24.389 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-10T19:59:24.418 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:59:24.455 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:24.925 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-10T19:59:24.941 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-10T19:59:24.974 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-10T19:59:25.286 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-10T19:59:25.315 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-10T19:59:25.352 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-10T19:59:25.614 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package jq. 2024-09-10T19:59:25.642 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-10T19:59:25.680 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-10T19:59:25.908 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package socat. 2024-09-10T19:59:25.936 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-10T19:59:25.966 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-10T19:59:26.278 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package xmlstarlet. 2024-09-10T19:59:26.307 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-10T19:59:26.345 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-10T19:59:26.683 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-test. 2024-09-10T19:59:26.699 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:26.740 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-test (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:29.279 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-volume. 2024-09-10T19:59:29.308 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T19:59:29.345 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-volume (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:29.716 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-10T19:59:29.733 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:29.774 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs-dev (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:30.094 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-10T19:59:30.123 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-10T19:59:30.160 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-10T19:59:30.474 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-10T19:59:30.502 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-10T19:59:30.532 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-10T19:59:30.895 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package nvme-cli. 2024-09-10T19:59:30.923 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-10T19:59:30.961 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-10T19:59:31.358 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package pkg-config. 2024-09-10T19:59:31.386 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-10T19:59:31.424 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-10T19:59:31.677 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-10T19:59:31.706 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-10T19:59:31.744 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-10T19:59:32.172 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-10T19:59:32.201 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-10T19:59:32.239 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-10T19:59:32.542 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastescript. 2024-09-10T19:59:32.570 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-10T19:59:32.608 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-10T19:59:32.912 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pluggy. 2024-09-10T19:59:32.929 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-10T19:59:32.962 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-10T19:59:33.265 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-psutil. 2024-09-10T19:59:33.293 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-10T19:59:33.331 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-10T19:59:33.668 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-py. 2024-09-10T19:59:33.696 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-10T19:59:33.734 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-10T19:59:34.055 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pygments. 2024-09-10T19:59:34.083 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-10T19:59:34.121 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-10T19:59:34.609 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-10T19:59:34.637 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-10T19:59:34.675 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-10T19:59:35.003 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-toml. 2024-09-10T19:59:35.032 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-10T19:59:35.102 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-10T19:59:35.531 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pytest. 2024-09-10T19:59:35.560 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-10T19:59:35.597 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-10T19:59:35.943 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplejson. 2024-09-10T19:59:35.972 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-10T19:59:36.009 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-10T19:59:36.281 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-10T19:59:36.310 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-10T19:59:36.347 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-10T19:59:37.059 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package radosgw. 2024-09-10T19:59:37.088 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../117-radosgw_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:37.126 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking radosgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:38.062 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package rbd-fuse. 2024-09-10T19:59:38.079 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T19:59:38.120 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking rbd-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:38.418 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-10T19:59:38.667 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-10T19:59:38.768 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-10T19:59:39.018 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-10T19:59:39.374 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-10T19:59:39.802 INFO:teuthology.orchestra.run.smithi039.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-10T19:59:39.861 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:40.360 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user cephadm....done 2024-09-10T19:59:40.475 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-10T19:59:40.760 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-10T19:59:41.019 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-10T19:59:41.129 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-10T19:59:41.388 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-10T19:59:41.664 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-10T19:59:41.790 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-10T19:59:42.159 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-10T19:59:42.535 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-10T19:59:42.795 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-10T19:59:43.062 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-10T19:59:43.314 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-10T19:59:43.557 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-10T19:59:43.808 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-10T19:59:44.042 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-argparse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:44.293 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-10T19:59:44.411 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-10T19:59:44.537 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-10T19:59:44.663 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-10T19:59:44.789 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-10T19:59:45.057 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-10T19:59:45.465 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-10T19:59:45.663 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-10T19:59:45.727 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-10T19:59:45.852 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-10T19:59:46.220 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-10T19:59:46.346 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-10T19:59:46.648 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-10T19:59:46.915 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-10T19:59:47.216 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up zip (3.0-12build2) ... 2024-09-10T19:59:47.343 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-10T19:59:48.036 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-10T19:59:48.296 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-10T19:59:48.422 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-10T19:59:48.548 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-10T19:59:48.840 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-10T19:59:49.266 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-10T19:59:49.501 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-10T19:59:49.877 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-10T19:59:50.195 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-10T19:59:50.555 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-10T19:59:50.814 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-10T19:59:50.940 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T19:59:51.258 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-10T19:59:51.584 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-10T19:59:52.765 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-10T19:59:52.992 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T19:59:53.126 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-10T19:59:53.394 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-10T19:59:53.520 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-10T19:59:53.646 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-10T19:59:53.964 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-10T19:59:54.215 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-10T19:59:54.450 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T19:59:54.551 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-10T19:59:54.835 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-10T19:59:55.095 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-10T19:59:55.412 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-10T19:59:55.747 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-10T19:59:56.057 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-10T19:59:56.307 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-10T19:59:56.433 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-10T19:59:56.718 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-10T19:59:56.827 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-10T19:59:57.120 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-10T19:59:57.438 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-10T19:59:57.689 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-10T19:59:57.790 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-10T19:59:57.891 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-10T19:59:57.972 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-10T19:59:58.218 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-10T19:59:58.343 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-10T19:59:58.745 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-10T19:59:59.021 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-10T19:59:59.147 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-10T19:59:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T19:59:59.507 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-10T19:59:59.825 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-10T20:00:00.110 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-10T20:00:00.386 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-10T20:00:00.512 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-10T20:00:00.796 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-10T20:00:01.158 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-10T20:00:01.417 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-10T20:00:01.802 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-10T20:00:02.095 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-10T20:00:02.448 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-10T20:00:02.574 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librados2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:02.683 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:02.784 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-10T20:00:03.103 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:03.229 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libradosstriper1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:03.355 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-10T20:00:03.640 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librbd1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:03.766 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-modules-core (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:03.892 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:04.110 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T20:00:04.111 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T20:00:04.610 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs-dev (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:04.736 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rados (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:04.862 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librgw2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:04.979 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-10T20:00:06.225 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rbd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:06.326 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up rbd-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:06.427 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:06.544 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cephfs (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:06.670 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:07.213 INFO:teuthology.orchestra.run.smithi039.stdout:Adding group ceph....done 2024-09-10T20:00:07.573 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user ceph....done 2024-09-10T20:00:07.741 INFO:teuthology.orchestra.run.smithi039.stdout:Setting system user ceph properties....done 2024-09-10T20:00:07.761 INFO:teuthology.orchestra.run.smithi039.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-10T20:00:07.908 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-10T20:00:08.213 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-10T20:00:08.713 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-test (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:08.840 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up radosgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:09.324 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:00:09.324 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:00:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-base (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:10.215 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-10T20:00:10.668 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mds (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:10.864 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T20:00:10.864 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T20:00:11.441 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T20:00:11.662 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T20:00:12.166 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-osd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:12.438 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T20:00:12.438 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T20:00:12.943 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:13.052 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:13.204 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mon (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:13.428 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T20:00:13.428 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T20:00:13.930 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:14.081 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:14.207 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-dashboard (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:14.360 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-volume (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:00:14.702 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-10T20:00:14.795 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-10T20:00:14.939 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-10T20:00:16.073 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-10T20:00:16.705 INFO:teuthology.orchestra.run.smithi039.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-10T20:00:16.710 DEBUG:teuthology.parallel:result is None 2024-09-10T20:00:16.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:00:16.852 DEBUG:teuthology.orchestra.run.smithi039:> dpkg-query -W -f '${Version}' ceph 2024-09-10T20:00:16.872 INFO:teuthology.orchestra.run.smithi039.stdout:19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:00:16.872 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:00:16.873 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858-gbad1f558-1jammy is installed. 2024-09-10T20:00:16.875 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T20:00:16.875 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T20:00:16.875 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:00:16.928 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T20:00:16.928 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T20:00:16.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:00:16.982 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:00:17.038 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T20:00:17.039 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T20:00:17.039 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:00:17.095 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:00:17.151 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T20:00:17.151 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T20:00:17.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:00:17.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:00:17.263 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T20:00:17.276 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:00:17.277 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T20:00:17.277 INFO:tasks.workunit:timeout=3h 2024-09-10T20:00:17.277 INFO:tasks.workunit:cleanup=True 2024-09-10T20:00:17.277 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:00:17.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:00:17.311 INFO:teuthology.orchestra.run.smithi039.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T20:00:17.312 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:00:17.360 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T20:00:17.360 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T20:00:17.407 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:00:17.457 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T20:01:04.584 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T20:01:04.584 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.585 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-09-10T20:01:04.586 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.586 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T20:01:04.586 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-09-10T20:01:04.586 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:01:04.590 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T20:01:04.640 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-10T20:01:04.640 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T20:01:04.686 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-10T20:01:04.687 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-10T20:01:04.687 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-10T20:01:04.757 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:01:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:01:04.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/bad-inc-map 2024-09-10T20:01:04.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-10T20:01:04.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-10T20:01:04.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:51501 2024-09-10T20:01:04.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:51501 2024-09-10T20:01:04.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-10T20:01:04.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-10T20:01:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none ' 2024-09-10T20:01:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:51501 ' 2024-09-10T20:01:04.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-10T20:01:04.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-10T20:01:04.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:01:04.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/bad-inc-map 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/bad-inc-map 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:01:04.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:01:04.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:01:04.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:01:04.796 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:01:04.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:01:04.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:01:04.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:01:04.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:01:04.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:01:04.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:01:04.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:01:04.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:01:04.800 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:01:04.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:01:04.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:01:04.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-10T20:01:04.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:01:04.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:04.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:04.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21385 2024-09-10T20:01:04.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:01:04.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:01:04.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/bad-inc-map 2024-09-10T20:01:04.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:01:04.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:04.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:04.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.21385 2024-09-10T20:01:04.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:01:04.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/bad-inc-map 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/bad-inc-map/a 2024-09-10T20:01:04.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-10T20:01:05.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:01:05.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:01:05.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:01:05.361 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:01:05.361 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:05.361 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:05.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:01:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:01:05.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:01:05.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:01:05.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:01:05.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:01:05.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:01:05.557 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:01:05.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:01:05.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:01:05.557 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:01:05.558 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:01:05.558 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:05.558 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:05.558 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21385/ceph-mon.a.asok 2024-09-10T20:01:05.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:01:05.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21385/ceph-mon.a.asok config get fsid 2024-09-10T20:01:05.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:01:05.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:01:05.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:01:05.673 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:01:05.674 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.21385/ceph-mon.a.asok 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:01:05.675 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.21385/ceph-mon.a.asok config get mon_host 2024-09-10T20:01:05.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-10T20:01:05.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/bad-inc-map 2024-09-10T20:01:05.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:01:05.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:01:05.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:01:05.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/bad-inc-map/x 2024-09-10T20:01:05.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:01:06.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:01:06.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:01:06.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:01:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:01:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:06.678 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:06.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:01:06.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:01:06.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:01:06.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-10T20:01:06.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/0 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 ' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:01:06.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-10T20:01:06.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:01:06.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:01:06.824 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:01:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:01:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:06.824 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:01:06.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:01:06.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/0 2024-09-10T20:01:06.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:01:06.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dbd3c6f1-56aa-40d2-9b05-797d725eb11e 2024-09-10T20:01:06.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dbd3c6f1-56aa-40d2-9b05-797d725eb11e' 2024-09-10T20:01:06.827 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 dbd3c6f1-56aa-40d2-9b05-797d725eb11e 2024-09-10T20:01:06.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:01:06.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCpeBmDKmHMhAARnxLbvX7UT+toN9ZlI7yKw== 2024-09-10T20:01:06.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCpeBmDKmHMhAARnxLbvX7UT+toN9ZlI7yKw=="}' 2024-09-10T20:01:06.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dbd3c6f1-56aa-40d2-9b05-797d725eb11e -i td/bad-inc-map/0/new.json 2024-09-10T20:01:07.150 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:01:07.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/0/new.json 2024-09-10T20:01:07.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCpeBmDKmHMhAARnxLbvX7UT+toN9ZlI7yKw== --osd-uuid dbd3c6f1-56aa-40d2-9b05-797d725eb11e 2024-09-10T20:01:07.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:07.255+0000 7f4522c6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:07.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:07.259+0000 7f4522c6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:07.263+0000 7f4522c6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:07.263+0000 7f4522c6c7c0 -1 bdev(0x564f764a3c00 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:01:07.263 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:07.263+0000 7f4522c6c7c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-10T20:01:13.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-10T20:01:13.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:01:13.360 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:01:13.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:01:13.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:01:13.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:01:13.602 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:01:13.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:01:13.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:01:13.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:01:13.603 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:01:13.634 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:13.635+0000 7f53f656d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:13.636 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:13.635+0000 7f53f656d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:13.637 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:13.639+0000 7f53f656d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:01:13.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:01:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:01:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:01:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:01:13.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:13.821 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:01:13.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:01:13.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:13.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:14.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:15.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:15.054 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:01:15.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:15.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:01:15.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:15.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:15.183 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:15.183+0000 7f53f656d7c0 -1 Falling back to public interface 2024-09-10T20:01:15.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:16.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:16.283 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:01:16.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:01:16.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:16.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:16.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:17.150 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:17.151+0000 7f53f656d7c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:01:17.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:17.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:17.734 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:01:17.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:01:17.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:17.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:18.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:19.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:19.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:19.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:01:19.320 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:01:19.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:19.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:19.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:20.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:20.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:20.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:01:20.920 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:01:20.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:20.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:21.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:01:22.375 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:01:22.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:22.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:22.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:23.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:23.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:23.881 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:01:23.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:01:23.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:23.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:24.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:25.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:25.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:25.338 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:01:25.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:01:25.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:25.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:25.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:26.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:26.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:26.916 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:01:26.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:01:26.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:26.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:27.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:28.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:28.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:28.369 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:01:28.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:01:28.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:28.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:29.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:30.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:30.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:30.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:01:30.198 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:01:30.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:30.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:31.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:32.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:32.045 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:01:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:01:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:32.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:33.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:34.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:34.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:34.006 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:01:34.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:01:34.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:34.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:34.585 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:35.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:35.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:35.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:01:35.587 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:01:35.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:35.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:36.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:37.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:37.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:37.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:01:37.240 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:01:37.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:37.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:38.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:39.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:39.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:39.939 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:01:39.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:01:39.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:39.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:40.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:01:41.795 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:01:41.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:41.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:01:42.330 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2933952010,v1:127.0.0.1:6801/2933952010] [v2:127.0.0.1:6802/2933952010,v1:127.0.0.1:6803/2933952010] exists,up dbd3c6f1-56aa-40d2-9b05-797d725eb11e 2024-09-10T20:01:42.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:01:42.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/1 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 ' 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:01:42.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-10T20:01:42.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:01:42.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/1 2024-09-10T20:01:42.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:01:42.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=18e484a3-9e86-483f-8c40-ca7d74387dc1 2024-09-10T20:01:42.339 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 18e484a3-9e86-483f-8c40-ca7d74387dc1 2024-09-10T20:01:42.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 18e484a3-9e86-483f-8c40-ca7d74387dc1' 2024-09-10T20:01:42.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:01:42.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmpeBm3YoAFRAAPo4NK2qkIeYGacJ7daJyjw== 2024-09-10T20:01:42.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmpeBm3YoAFRAAPo4NK2qkIeYGacJ7daJyjw=="}' 2024-09-10T20:01:42.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 18e484a3-9e86-483f-8c40-ca7d74387dc1 -i td/bad-inc-map/1/new.json 2024-09-10T20:01:42.905 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:01:42.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/1/new.json 2024-09-10T20:01:42.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmpeBm3YoAFRAAPo4NK2qkIeYGacJ7daJyjw== --osd-uuid 18e484a3-9e86-483f-8c40-ca7d74387dc1 2024-09-10T20:01:42.957 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:42.955+0000 7f6a585657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:42.959 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:42.959+0000 7f6a585657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:42.960 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:42.959+0000 7f6a585657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:42.959+0000 7f6a585657c0 -1 bdev(0x556edc993c00 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:01:42.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:42.959+0000 7f6a585657c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-10T20:01:49.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-10T20:01:49.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:01:49.314 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:01:49.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:01:49.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:01:49.904 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:01:49.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:01:49.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:01:49.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:01:49.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:01:49.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:01:49.935 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:49.935+0000 7f963210e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:49.946 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:49.947+0000 7f963210e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:49.947 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:49.947+0000 7f963210e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:50.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:01:50.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:01:50.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:01:50.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:01:50.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:01:50.393 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:01:50.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:50.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:01:50.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:50.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:50.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:51.676 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:51.675+0000 7f963210e7c0 -1 Falling back to public interface 2024-09-10T20:01:51.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:51.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:51.886 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:01:51.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:01:51.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:51.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:52.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:53.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:53.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:53.354 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:01:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:01:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:53.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:53.782 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:01:53.783+0000 7f963210e7c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:01:53.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:54.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:54.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:54.793 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:01:54.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:01:54.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:54.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:55.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:56.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:56.241 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:01:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:01:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:56.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:56.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:57.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:57.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:57.710 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:01:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:01:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:57.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:58.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:59.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:59.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:59.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:01:59.250 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:01:59.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:59.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:01:59.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:00.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:00.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:00.866 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:02:00.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:02:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:01.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:02.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:02.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:02.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:02:02.308 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:02:02.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:02.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:02.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:03.761 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:03.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:03.762 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:02:03.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:02:03.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:03.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:04.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:05.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:05.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:05.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:02:05.289 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:02:05.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:05.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:05.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:06.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:06.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:06.922 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:02:06.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:02:06.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:06.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:07.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:08.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:08.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:08.429 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:02:08.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:02:08.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:08.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:09.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:10.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:10.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:10.133 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:02:10.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:02:10.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:10.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:11.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:12.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:12.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:12.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:02:12.233 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:02:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:12.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:13.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:14.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:14.255 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:02:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:02:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:14.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:14.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:15.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:15.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:02:15.986 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:02:15.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:15.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:16.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:17.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:17.917 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:02:17.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:17.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:02:17.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:17.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:02:18.418 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3406224731,v1:127.0.0.1:6811/3406224731] [v2:127.0.0.1:6812/3406224731,v1:127.0.0.1:6813/3406224731] exists,up 18e484a3-9e86-483f-8c40-ca7d74387dc1 2024-09-10T20:02:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:02:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:02:18.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/bad-inc-map 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/bad-inc-map/2 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 ' 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-10T20:02:18.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:02:18.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:02:18.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/bad-inc-map/2 2024-09-10T20:02:18.425 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:02:18.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=734ae449-945f-4767-beba-8293054dc665 2024-09-10T20:02:18.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 734ae449-945f-4767-beba-8293054dc665' 2024-09-10T20:02:18.425 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 734ae449-945f-4767-beba-8293054dc665 2024-09-10T20:02:18.426 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:02:18.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKpeBm5goqGhAAVGlCymtueFOA5JR2zbmotw== 2024-09-10T20:02:18.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKpeBm5goqGhAAVGlCymtueFOA5JR2zbmotw=="}' 2024-09-10T20:02:18.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 734ae449-945f-4767-beba-8293054dc665 -i td/bad-inc-map/2/new.json 2024-09-10T20:02:18.942 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:02:18.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/bad-inc-map/2/new.json 2024-09-10T20:02:18.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKpeBm5goqGhAAVGlCymtueFOA5JR2zbmotw== --osd-uuid 734ae449-945f-4767-beba-8293054dc665 2024-09-10T20:02:18.988 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:18.980+0000 7f49068ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:02:18.990 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:18.984+0000 7f49068ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:02:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:18.984+0000 7f49068ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:02:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:18.984+0000 7f49068ab7c0 -1 bdev(0x5583168bfc00 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:02:18.991 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:18.984+0000 7f49068ab7c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-10T20:02:25.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-10T20:02:25.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:02:25.203 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:02:25.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:02:25.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:02:25.813 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:02:25.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:02:25.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=8151c832-554e-402a-8153-e92938ff1262 --auth-supported=none --mon-host=127.0.0.1:51501 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21385/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:02:25.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:02:25.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:02:25.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:02:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:25.839+0000 7f4e597657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:02:25.882 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:25.879+0000 7f4e597657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:02:25.883 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:25.879+0000 7f4e597657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:02:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:02:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:02:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:02:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:02:26.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:02:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:26.287 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:02:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:02:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:26.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:26.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:27.131 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:27.127+0000 7f4e597657c0 -1 Falling back to public interface 2024-09-10T20:02:27.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:27.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:27.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:02:27.756 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:02:27.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:27.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:28.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:28.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:02:28.891+0000 7f4e597657c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:02:29.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:29.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:29.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:02:29.230 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:02:29.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:29.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:29.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:30.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:02:30.720 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:02:30.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:30.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:31.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:32.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:32.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:32.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:02:32.318 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:02:32.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:32.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:32.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:02:33.794 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:02:33.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:33.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:34.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:35.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:35.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:35.281 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:02:35.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:02:35.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:35.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:35.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:36.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:36.912 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:36.913 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:02:36.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:02:36.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:36.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:37.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:38.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:38.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:38.356 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:02:38.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:02:38.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:38.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:38.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:39.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:39.832 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:02:39.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:39.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:02:39.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:39.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:41.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:41.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:41.327 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:02:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:02:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:41.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:42.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:43.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:43.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:43.840 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:02:43.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:02:43.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:43.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:44.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:45.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:45.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:45.850 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:02:45.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:02:45.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:45.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:47.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:48.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:48.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:48.236 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:02:48.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:02:48.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:48.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:48.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:49.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:49.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:49.985 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:02:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:02:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:49.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:51.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:52.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:52.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:52.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:02:52.154 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:02:52.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:52.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:02:52.617 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3480436854,v1:127.0.0.1:6819/3480436854] [v2:127.0.0.1:6820/3480436854,v1:127.0.0.1:6821/3480436854] exists,up 734ae449-945f-4767-beba-8293054dc665 2024-09-10T20:02:52.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:02:52.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:02:52.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:02:52.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-10T20:02:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-10T20:02:53.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 8 2024-09-10T20:02:53.791 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-10T20:02:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:02:54.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-10T20:02:56.231 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 1 2024-09-10T20:02:56.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-10T20:02:57.469 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-10T20:02:57.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-10T20:03:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-10T20:03:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-10T20:03:02.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:02.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:03:02.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:02.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:02.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-10T20:03:02.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:02.498 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:03:02.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:02.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:02.988 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2933952010,v1:127.0.0.1:6801/2933952010] [v2:127.0.0.1:6802/2933952010,v1:127.0.0.1:6803/2933952010] exists,up dbd3c6f1-56aa-40d2-9b05-797d725eb11e 2024-09-10T20:03:02.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-10T20:03:02.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:02.991 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:03:02.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:02.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:03.446 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3406224731,v1:127.0.0.1:6811/3406224731] [v2:127.0.0.1:6812/3406224731,v1:127.0.0.1:6813/3406224731] exists,up 18e484a3-9e86-483f-8c40-ca7d74387dc1 2024-09-10T20:03:03.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 10 )) 2024-09-10T20:03:03.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:03.449 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:03:03.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:03.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:03:03.964 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3480436854,v1:127.0.0.1:6819/3480436854] [v2:127.0.0.1:6820/3480436854,v1:127.0.0.1:6821/3480436854] exists,up 734ae449-945f-4767-beba-8293054dc665 2024-09-10T20:03:03.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:03:03.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:03:03.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:03:03.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-10T20:03:03.966 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:53.658+0000 7f4e4c70d640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.966 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:54.146+0000 7f4e4c70d640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:54.486+0000 7f4e4c70d640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:55.762+0000 7f4e4a709640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:55.962+0000 7f4e4a709640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:56.274+0000 7f4e4c70d640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:56.686+0000 7f4e4c70d640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:57.102+0000 7f4e4c70d640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:57.518+0000 7f4e4c70d640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:57.930+0000 7f4e4c70d640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:58.346+0000 7f4e4c70d640 -1 osd.2 27 handle_osd_map injecting map crc failure 2024-09-10T20:03:03.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:53.658+0000 7f4e4c70d640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:54.146+0000 7f4e4c70d640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:54.486+0000 7f4e4c70d640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:55.762+0000 7f4e4a709640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:55.962+0000 7f4e4a709640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:56.274+0000 7f4e4c70d640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:56.690+0000 7f4e4c70d640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:57.106+0000 7f4e4c70d640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:57.522+0000 7f4e4c70d640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:57.930+0000 7f4e4c70d640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-09-10T20:03:03.969 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:02:58.346+0000 7f4e4c70d640 10 osd.2 27 handle_osd_map bailing because last < start (27<28) 2024-09-10T20:03:03.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-10T20:03:03.970 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-10T20:03:03.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-10T20:03:03.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=foo 2024-09-10T20:03:03.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-10T20:03:04.559 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' does not exist 2024-09-10T20:03:04.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-10T20:03:04.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:04.578 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:04.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:04.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:04.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:05.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:05.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:05.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:05.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:05.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:05.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:03:05.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:03:05.900 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:03:05.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:03:05.901 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:03:05.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:03:05.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:05.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:03:05.903 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:03:05.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:05.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:03:05.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:03:05.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-10T20:03:05.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:03:05.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:05.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:03:05.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21385 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/bad-inc-map 0 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/bad-inc-map 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/bad-inc-map KILL 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:05.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:05.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:05.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:05.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:03:05.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:03:05.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:03:05.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:03:05.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:03:05.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:03:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:03:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:03:05.928 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:05.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:03:05.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:03:05.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/bad-inc-map 2024-09-10T20:03:05.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:03:05.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:05.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.21385 2024-09-10T20:03:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.21385 2024-09-10T20:03:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:03:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:03:05.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:03:05.933 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:03:05.933 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:03:05.983 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-10T20:03:05.984 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-10T20:03:06.035 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:03:06.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/divergent-priors 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-10T20:03:06.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-10T20:03:06.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-10T20:03:06.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-10T20:03:06.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-10T20:03:06.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-10T20:03:06.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-10T20:03:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-10T20:03:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-10T20:03:06.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-10T20:03:06.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-10T20:03:06.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-10T20:03:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none ' 2024-09-10T20:03:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-10T20:03:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-10T20:03:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-10T20:03:06.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:03:06.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-10T20:03:06.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:03:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-10T20:03:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_divergent_2 2024-09-10T20:03:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_divergent_3 2024-09-10T20:03:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_divergent_ec' 2024-09-10T20:03:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-10T20:03:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:06.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:06.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:03:06.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:03:06.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:03:06.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:03:06.058 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:03:06.059 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:03:06.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:06.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:03:06.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:03:06.060 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:06.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:03:06.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:03:06.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:03:06.063 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:03:06.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:06.064 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:06.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:03:06.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:03:06.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:03:06.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-10T20:03:06.066 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:03:06.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:06.067 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:06.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27146 2024-09-10T20:03:06.068 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:03:06.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:03:06.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:03:06.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:03:06.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-10T20:03:06.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-10T20:03:06.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-10T20:03:06.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-10T20:03:06.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-10T20:03:06.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-10T20:03:06.070 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-10T20:03:06.070 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:03:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-10T20:03:06.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-10T20:03:06.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:03:06.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:06.535 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:06.535 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:06.535 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:06.535 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:06.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:03:06.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:03:06.756 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:03:06.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:03:06.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:03:06.757 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:03:06.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:03:06.758 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:03:06.758 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:03:06.758 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:03:06.759 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:03:06.759 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:03:06.759 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:06.760 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:06.760 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:03:06.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:03:06.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get fsid 2024-09-10T20:03:06.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:03:06.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:03:06.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:03:06.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:03:06.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:03:06.880 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:03:06.880 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:03:06.880 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:03:06.880 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:03:06.881 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:06.881 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:06.881 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:03:06.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:03:06.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get mon_host 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-10T20:03:06.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:03:07.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:03:07.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:07.932 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:07.933 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:07.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:03:07.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:03:07.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:03:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:03:07.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:03:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:03:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:03:07.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:03:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=497c7066-97a5-4ff8-bbc5-fa078793cc71 2024-09-10T20:03:07.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 497c7066-97a5-4ff8-bbc5-fa078793cc71' 2024-09-10T20:03:07.999 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 497c7066-97a5-4ff8-bbc5-fa078793cc71 2024-09-10T20:03:07.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:03:08.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8peBmnoYaARAAvobHJpvA3Vmt3E+RtzRiHQ== 2024-09-10T20:03:08.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8peBmnoYaARAAvobHJpvA3Vmt3E+RtzRiHQ=="}' 2024-09-10T20:03:08.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 497c7066-97a5-4ff8-bbc5-fa078793cc71 -i td/divergent-priors/0/new.json 2024-09-10T20:03:08.333 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:03:08.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-10T20:03:08.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8peBmnoYaARAAvobHJpvA3Vmt3E+RtzRiHQ== --osd-uuid 497c7066-97a5-4ff8-bbc5-fa078793cc71 2024-09-10T20:03:08.395 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:08.390+0000 7f855a0467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:08.402 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:08.398+0000 7f855a0467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:08.403 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:08.398+0000 7f855a0467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:08.398+0000 7f855a0467c0 -1 bdev(0x56459337bc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:03:08.404 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:08.398+0000 7f855a0467c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-10T20:03:14.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-10T20:03:14.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:03:14.572 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:03:14.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:03:14.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:03:15.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:03:15.197 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:03:15.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:03:15.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:03:15.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:03:15.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:03:15.240 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:15.233+0000 7efd9f9937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:15.251 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:15.245+0000 7efd9f9937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:15.252 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:15.245+0000 7efd9f9937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:15.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:03:15.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:15.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:03:15.680 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:03:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:15.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:16.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:16.464 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:16.457+0000 7efd9f9937c0 -1 Falling back to public interface 2024-09-10T20:03:17.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:17.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:17.183 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:03:17.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:03:17.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:17.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:17.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:18.151 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:18.145+0000 7efd9f9937c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:03:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:18.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:18.639 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:03:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:03:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:18.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:19.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:20.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:20.274 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:03:20.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:03:20.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:20.275 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:20.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:21.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:21.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:21.743 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:03:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:03:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:21.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:22.319 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:23.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:23.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:23.322 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:03:23.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:03:23.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:23.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:23.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:24.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:24.810 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:03:24.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:24.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:03:24.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:24.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:25.301 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:26.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:26.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:26.304 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:03:26.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:03:26.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:26.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:26.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:27.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:27.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:27.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:03:27.795 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:03:27.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:27.795 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:28.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:29.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:29.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:29.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:03:29.337 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:03:29.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:29.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:29.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:30.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:30.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:30.793 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:03:30.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:03:30.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:30.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:31.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:32.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:32.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:32.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:03:32.327 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:03:32.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:32.328 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:32.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:33.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:33.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:33.940 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:03:33.940 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:03:33.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:33.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:34.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:35.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:35.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:35.748 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:03:35.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:03:35.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:35.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:36.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:37.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:37.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:37.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:03:37.582 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:03:37.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:37.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:38.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:39.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:39.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:39.097 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:03:39.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:03:39.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:39.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:40.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:41.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:41.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:41.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:03:41.410 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:03:41.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:41.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:41.927 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3871433343,v1:127.0.0.1:6803/3871433343] [v2:127.0.0.1:6804/3871433343,v1:127.0.0.1:6805/3871433343] exists,up 497c7066-97a5-4ff8-bbc5-fa078793cc71 2024-09-10T20:03:41.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:03:41.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:03:41.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:41.931 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:03:41.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:03:41.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:03:41.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:03:41.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:03:41.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 2024-09-10T20:03:41.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 cdd2bd3f-14b2-48a4-bc4b-6987c79dd657' 2024-09-10T20:03:41.935 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 2024-09-10T20:03:41.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:03:41.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAdpuBmqQrnOBAAxbyRamjUqtgyNzShF8S0wg== 2024-09-10T20:03:41.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAdpuBmqQrnOBAAxbyRamjUqtgyNzShF8S0wg=="}' 2024-09-10T20:03:41.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 -i td/divergent-priors/1/new.json 2024-09-10T20:03:42.529 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:03:42.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-10T20:03:42.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdpuBmqQrnOBAAxbyRamjUqtgyNzShF8S0wg== --osd-uuid cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 2024-09-10T20:03:42.591 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:42.584+0000 7f2775b857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:42.610 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:42.604+0000 7f2775b857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:42.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:42.604+0000 7f2775b857c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:42.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:42.604+0000 7f2775b857c0 -1 bdev(0x55c98d1ebc00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:03:42.612 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:42.604+0000 7f2775b857c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-10T20:03:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-10T20:03:48.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:03:48.770 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:03:48.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:03:48.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:03:49.418 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:03:49.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:03:49.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:03:49.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:03:49.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:03:49.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:03:49.460 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:49.452+0000 7f501af077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:49.462 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:49.456+0000 7f501af077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:49.463 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:49.456+0000 7f501af077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:03:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:03:49.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:49.872 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:03:49.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:49.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:49.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:49.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:49.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:50.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:51.212 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:51.204+0000 7f501af077c0 -1 Falling back to public interface 2024-09-10T20:03:51.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:51.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:51.353 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:03:51.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:03:51.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:51.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:51.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:52.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:52.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:52.847 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:03:52.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:03:52.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:52.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:52.976 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:03:52.968+0000 7f501af077c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:03:53.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:54.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:54.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:54.443 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:03:54.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:03:54.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:54.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:54.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:55.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:55.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:56.000 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:03:56.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:03:56.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:56.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:56.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:57.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:57.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:57.490 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:03:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:03:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:57.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:58.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:59.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:59.064 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:03:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:03:59.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:59.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:59.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:00.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:00.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:00.535 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:04:00.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:04:00.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:00.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:01.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:02.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:02.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:02.062 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:04:02.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:04:02.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:02.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:02.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:03.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:03.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:03.534 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:04:03.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:04:03.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:03.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:04.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:05.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:05.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:05.075 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:04:05.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:04:05.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:05.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:05.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:06.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:06.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:06.545 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:04:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:04:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:06.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:07.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:08.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:08.408 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:08.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:09.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:10.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:10.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:10.320 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:04:10.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:04:10.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:10.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:11.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:12.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:12.645 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:04:12.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:12.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:04:12.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:12.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:13.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:14.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:14.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:14.411 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:04:14.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:04:14.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:14.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:15.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:16.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:16.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:16.172 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:04:16.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:04:16.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:16.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:16.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:17.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:17.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:17.674 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:04:17.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:04:17.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:17.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:18.160 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4242344907,v1:127.0.0.1:6811/4242344907] [v2:127.0.0.1:6812/4242344907,v1:127.0.0.1:6813/4242344907] exists,up cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:04:18.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:04:18.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:04:18.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:04:18.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=da610e6e-6058-42ca-b517-21c1e0ba0f0a 2024-09-10T20:04:18.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 da610e6e-6058-42ca-b517-21c1e0ba0f0a' 2024-09-10T20:04:18.168 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 da610e6e-6058-42ca-b517-21c1e0ba0f0a 2024-09-10T20:04:18.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:04:18.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCpuBmXy0JCxAAzag0Rou6brnKAhYot4nomw== 2024-09-10T20:04:18.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCpuBmXy0JCxAAzag0Rou6brnKAhYot4nomw=="}' 2024-09-10T20:04:18.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new da610e6e-6058-42ca-b517-21c1e0ba0f0a -i td/divergent-priors/2/new.json 2024-09-10T20:04:18.729 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:04:18.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-10T20:04:18.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCpuBmXy0JCxAAzag0Rou6brnKAhYot4nomw== --osd-uuid da610e6e-6058-42ca-b517-21c1e0ba0f0a 2024-09-10T20:04:18.795 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:18.788+0000 7f0870d787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:18.809 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:18.804+0000 7f0870d787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:18.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:18.804+0000 7f0870d787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:18.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:18.804+0000 7f0870d787c0 -1 bdev(0x55df9c4afc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:04:18.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:18.804+0000 7f0870d787c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-10T20:04:24.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-10T20:04:24.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:04:24.649 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:04:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:04:24.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:04:25.279 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:04:25.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:04:25.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:04:25.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:04:25.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:04:25.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:04:25.322 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:25.316+0000 7f65b19287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:25.339 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:25.332+0000 7f65b19287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:25.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:25.332+0000 7f65b19287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:25.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:04:25.735 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:04:25.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:04:25.736 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:04:25.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:04:25.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:04:25.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:25.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:04:25.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:25.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:26.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:26.580 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:26.572+0000 7f65b19287c0 -1 Falling back to public interface 2024-09-10T20:04:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:27.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:27.239 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:04:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:04:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:27.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:27.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:28.255 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:04:28.248+0000 7f65b19287c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:04:28.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:28.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:28.740 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:04:28.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:04:28.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:28.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:29.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:30.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:30.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:30.471 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:04:30.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:04:30.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:30.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:31.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:32.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:32.040 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:04:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:04:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:32.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:32.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:33.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:33.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:33.514 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:04:33.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:04:33.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:33.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:34.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:35.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:35.039 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:04:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:04:35.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:35.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:35.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:36.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:36.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:36.655 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:04:36.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:04:36.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:36.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:37.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:38.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:38.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:38.201 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:04:38.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:04:38.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:38.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:38.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:39.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:39.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:39.668 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:04:39.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:04:39.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:39.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:40.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:41.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:41.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:41.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:04:41.186 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:04:41.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:41.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:41.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:42.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:42.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:42.891 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:04:42.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:04:42.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:42.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:43.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:44.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:44.591 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:04:44.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:44.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:04:44.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:44.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:45.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:46.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:46.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:46.283 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:04:46.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:04:46.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:46.283 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:47.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:48.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:48.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:48.023 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:04:48.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:04:48.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:48.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:48.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:49.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:49.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:49.767 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:04:49.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:04:49.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:49.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:50.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:51.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:51.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:51.562 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:04:51.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:04:51.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:51.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:52.028 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3916415163,v1:127.0.0.1:6819/3916415163] [v2:127.0.0.1:6820/3916415163,v1:127.0.0.1:6821/3916415163] exists,up da610e6e-6058-42ca-b517-21c1e0ba0f0a 2024-09-10T20:04:52.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:04:52.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:04:52.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:04:52.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-10T20:04:52.773 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-10T20:04:52.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-10T20:04:53.653 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-10T20:04:53.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-10T20:04:54.456 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-10T20:04:54.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-10T20:04:54.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T20:04:55.151 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-10T20:04:55.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:04:56.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-10T20:04:57.192 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-10T20:04:57.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-10T20:04:58.007 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-10T20:04:58.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-10T20:04:58.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:04:58.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:04:58.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:04:58.591 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:04:58.591 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:04:58.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:04:58.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:04:58.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:04:58.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-10T20:04:58.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-10T20:04:58.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-10T20:04:58.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:04:58.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:04:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-10T20:04:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-10T20:04:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672974' 2024-09-10T20:04:58.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:04:58.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:04:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509447 2024-09-10T20:04:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509447 2024-09-10T20:04:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672974 2-64424509447' 2024-09-10T20:04:59.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:04:59.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-10T20:04:59.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:04:59.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:04:59.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-10T20:04:59.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:04:59.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-10T20:04:59.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-10T20:04:59.103 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836503 2024-09-10T20:04:59.103 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:04:59.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-10T20:04:59.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:04:59.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-10T20:04:59.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:04:59.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:04:59.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-10T20:04:59.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:04:59.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-10T20:04:59.594 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-09-10T20:04:59.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-10T20:04:59.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:05:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-10T20:05:00.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:00.101 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509447 2024-09-10T20:05:00.102 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:00.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:05:00.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:00.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509447 2024-09-10T20:05:00.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509447 2024-09-10T20:05:00.105 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509447 2024-09-10T20:05:00.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-10T20:05:00.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:00.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509445 -lt 64424509447 2024-09-10T20:05:00.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:05:01.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:05:01.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:02.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-10T20:05:02.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-10T20:05:02.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:05:02.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:05:02.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:05:02.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:05:02.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:05:02.015 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:05:02.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:05:02.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:05:02.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:05:02.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:05:02.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:05:02.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:05:02.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:05:02.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:05:02.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:05:02.605 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:05:02.605 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:05:02.606 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:05:02.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:05:02.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:02.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:05:02.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T20:05:02.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T20:05:02.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T20:05:02.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:02.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:05:02.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-10T20:05:02.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-10T20:05:02.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672977' 2024-09-10T20:05:02.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:02.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:05:03.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509450 2024-09-10T20:05:03.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509450 2024-09-10T20:05:03.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672977 2-64424509450' 2024-09-10T20:05:03.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:03.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T20:05:03.128 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:03.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:05:03.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T20:05:03.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:03.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T20:05:03.130 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836506 2024-09-10T20:05:03.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T20:05:03.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:05:03.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T20:05:03.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:03.606 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:03.607 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-10T20:05:03.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:05:03.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-10T20:05:03.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:03.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-10T20:05:03.610 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672977 2024-09-10T20:05:03.610 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-10T20:05:03.611 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:05:04.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-10T20:05:04.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:04.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509450 2024-09-10T20:05:04.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:04.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:05:04.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509450 2024-09-10T20:05:04.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:04.097 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509450 2024-09-10T20:05:04.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509450 2024-09-10T20:05:04.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-10T20:05:04.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:04.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509450 2024-09-10T20:05:04.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:05:05.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:05:05.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:06.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-10T20:05:06.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:05:06.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:05:06.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:05:06.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:05:06.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:05:06.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:05:06.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:05:06.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:05:06.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:05:06.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:05:06.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:05:07.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:05:07.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:05:07.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:05:07.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:05:07.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:05:07.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:05:07.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:05:07.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-10T20:05:07.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-10T20:05:08.060 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:05:08.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-10T20:05:08.150 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-10T20:05:08.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-10T20:05:08.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-10T20:05:08.517 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:05:08.517 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:04:56.582445+0000 0'0 29:20 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:59:29.018775+0000 0 0 2024-09-10T20:05:08.517 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:05:08.517 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:05:08.517 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-10T20:05:08.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-10T20:05:08.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-10T20:05:08.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-10T20:05:08.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-10T20:05:08.536 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-10T20:05:08.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-10T20:05:08.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:08.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-10T20:05:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:08.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-10T20:05:08.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:08.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-10T20:05:09.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:09.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-10T20:05:09.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:09.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-10T20:05:09.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:09.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-10T20:05:09.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:09.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-10T20:05:09.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:09.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-10T20:05:09.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:09.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-10T20:05:10.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:10.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-10T20:05:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-10T20:05:10.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:10.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-10T20:05:10.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:10.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-10T20:05:10.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:10.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-10T20:05:11.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:11.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-10T20:05:11.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:11.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-10T20:05:11.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:11.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-10T20:05:11.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:11.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-10T20:05:11.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:11.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-10T20:05:11.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:11.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-10T20:05:12.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:12.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-10T20:05:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:12.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-10T20:05:12.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:12.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-10T20:05:12.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:12.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-10T20:05:12.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:12.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-10T20:05:12.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:12.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-10T20:05:13.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:13.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-10T20:05:13.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:13.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-10T20:05:13.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:13.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-10T20:05:13.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:13.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-10T20:05:13.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:13.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-10T20:05:14.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:14.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-10T20:05:14.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:14.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-10T20:05:14.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:14.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-10T20:05:14.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:14.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-10T20:05:14.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:14.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-10T20:05:14.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:14.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-10T20:05:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:15.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-10T20:05:15.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:15.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-10T20:05:15.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:15.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-10T20:05:15.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:15.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-10T20:05:15.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:15.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-10T20:05:15.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:15.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-10T20:05:16.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:16.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-10T20:05:16.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:16.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-10T20:05:16.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:16.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-10T20:05:16.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:16.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-10T20:05:16.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:16.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-10T20:05:16.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:16.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-10T20:05:17.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:17.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-10T20:05:17.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:17.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-10T20:05:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:17.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-10T20:05:17.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:17.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-10T20:05:17.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:17.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-10T20:05:17.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:17.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-10T20:05:18.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:18.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-10T20:05:18.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:18.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-10T20:05:18.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:18.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-10T20:05:18.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:18.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-10T20:05:18.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:18.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-10T20:05:18.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:18.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-10T20:05:19.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:19.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-10T20:05:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:19.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-10T20:05:19.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:19.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-10T20:05:19.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:19.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-10T20:05:19.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:19.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-10T20:05:19.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:19.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-10T20:05:20.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:20.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-10T20:05:20.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:20.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-10T20:05:20.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:20.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-10T20:05:20.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:20.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-10T20:05:20.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:20.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-10T20:05:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:20.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-10T20:05:21.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:21.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-10T20:05:21.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:21.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-10T20:05:21.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:21.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-10T20:05:21.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:21.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-10T20:05:21.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:21.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-10T20:05:22.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:22.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-10T20:05:22.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:22.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-10T20:05:22.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:22.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-10T20:05:22.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:22.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-10T20:05:22.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:22.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-10T20:05:22.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:22.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-10T20:05:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:23.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-10T20:05:23.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:23.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-10T20:05:23.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:23.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-10T20:05:23.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:23.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-10T20:05:23.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:23.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-10T20:05:24.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:24.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-10T20:05:24.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:24.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-10T20:05:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-10T20:05:24.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:24.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-10T20:05:24.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:24.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-10T20:05:24.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:24.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-10T20:05:25.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:25.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-10T20:05:25.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:25.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-10T20:05:25.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:25.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-10T20:05:25.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:25.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-10T20:05:25.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-10T20:05:25.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-10T20:05:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:05:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-10T20:05:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:05:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:05:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:05:25.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:05:25.903 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:05:25.904 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:05:25.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:05:25.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:05:25.904 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:05:25.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:05:25.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:05:25.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:05:25.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:05:25.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:05:25.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:05:26.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:05:26.476 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:05:26.477 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:05:26.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:05:26.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:26.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:05:26.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-10T20:05:26.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-10T20:05:26.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-10T20:05:26.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:26.663 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:05:26.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-10T20:05:26.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-10T20:05:26.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-09-10T20:05:26.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:26.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:05:27.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509456 2024-09-10T20:05:27.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509456 2024-09-10T20:05:27.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509456' 2024-09-10T20:05:27.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:27.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-10T20:05:27.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:27.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:05:27.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-10T20:05:27.004 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:27.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-10T20:05:27.005 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836512 2024-09-10T20:05:27.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-10T20:05:27.005 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:05:27.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836513 -lt 21474836512 2024-09-10T20:05:27.512 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:27.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:27.513 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-10T20:05:27.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:05:27.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-10T20:05:27.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:27.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-10T20:05:27.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-10T20:05:27.517 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672983 2024-09-10T20:05:27.517 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:05:28.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-10T20:05:28.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:28.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509456 2024-09-10T20:05:28.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:28.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:05:28.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509456 2024-09-10T20:05:28.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:28.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509456 2024-09-10T20:05:28.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-10T20:05:28.020 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509456 2024-09-10T20:05:28.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:28.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509456 2024-09-10T20:05:28.549 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:05:29.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:05:29.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:30.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509457 -lt 64424509456 2024-09-10T20:05:30.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:05:30.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:05:30.036 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:05:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:05:30.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:05:30.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:05:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:05:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:05:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:05:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:05:30.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:05:31.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:05:31.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:05:31.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:05:31.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:05:31.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:05:31.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:05:31.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:05:31.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-10T20:05:31.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-10T20:05:31.644 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-10T20:05:31.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-10T20:05:31.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:05:32.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-10T20:05:32.131 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-10T20:05:32.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-10T20:05:32.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-10T20:05:32.554 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:05:32.554 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-10T20:04:56.582445+0000 29'100 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:59:29.018775+0000 0 0 2024-09-10T20:05:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:05:32.555 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:05:32.555 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:05:32.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:32.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-10T20:05:32.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:05:32.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:05:32.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:05:32.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:05:32.573 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:05:32.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:05:32.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-10T20:05:32.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-10T20:05:32.673 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:05:32.673 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:05:32.673 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:05:32.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:32.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-10T20:05:32.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:05:32.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:05:32.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:05:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:05:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:05:32.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:05:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-10T20:05:32.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-10T20:05:32.785 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:05:32.785 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:05:32.785 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:05:32.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-10T20:05:32.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-10T20:05:32.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-10T20:05:32.798 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-10T20:05:32.799 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent object 2024-09-10T20:05:32.799 INFO:tasks.workunit.client.0.smithi039.stdout:create missing divergent object 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2402: inject_eio: local pooltype=rep 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2403: inject_eio: shift 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: inject_eio: local which=data 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: inject_eio: shift 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: inject_eio: local poolname=test 2024-09-10T20:05:32.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: inject_eio: shift 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: inject_eio: local objname=existing_99 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: inject_eio: shift 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local dir=td/divergent-priors 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local shard_id=0 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: get_osds test existing_99 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=test 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=existing_99 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map test existing_99 2024-09-10T20:05:32.802 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2024-09-10T20:05:33.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:0 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: initial_osds=('1' '0' '2') 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local -a initial_osds 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local osd_id=1 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: '[' rep '!=' ec ']' 2024-09-10T20:05:33.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shard_id= 2024-09-10T20:05:33.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: cat td/divergent-priors/1/type 2024-09-10T20:05:33.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: type=bluestore 2024-09-10T20:05:33.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-10T20:05:33.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1152: set_config: local daemon=osd 2024-09-10T20:05:33.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1153: set_config: local id=1 2024-09-10T20:05:33.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1154: set_config: local config=bluestore_debug_inject_read_err 2024-09-10T20:05:33.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1155: set_config: local value=true 2024-09-10T20:05:33.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: jq 'has("success")' 2024-09-10T20:05:33.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: get_asok_path osd.1 2024-09-10T20:05:33.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:05:33.338 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:05:33.338 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:05:33.339 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:05:33.339 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:05:33.339 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:05:33.339 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.27146/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-10T20:05:33.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1159: set_config: test true == true 2024-09-10T20:05:33.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: local loop=0 2024-09-10T20:05:33.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: grep -q Invalid 2024-09-10T20:05:33.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_asok_path osd.1 2024-09-10T20:05:33.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:05:33.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:05:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:05:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:05:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:05:33.469 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:05:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: CEPH_ARGS= 2024-09-10T20:05:33.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok injectdataerr test existing_99 2024-09-10T20:05:33.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-10T20:05:33.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-10T20:05:43.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-10T20:05:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-10T20:05:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-10T20:05:43.615 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-10T20:05:43.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-10T20:05:44.046 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:05:44.046 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-09-10T20:05:33.744490+0000 29'101 29:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:59:29.018775+0000 0 0 2024-09-10T20:05:44.046 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:05:44.046 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:05:44.046 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:05:44.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-10T20:05:44.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:05:44.062 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:05:44.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:05:44.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:05:44.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:05:44.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:05:44.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-10T20:05:44.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-10T20:05:44.730 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:05:44.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-10T20:05:44.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-10T20:05:45.347 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:05:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-10T20:05:45.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-10T20:05:45.942 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:05:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:45.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-10T20:05:46.499 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-10T20:05:46.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:46.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-10T20:05:47.094 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-10T20:05:47.111 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-10T20:05:47.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-10T20:05:47.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-10T20:05:47.539 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:05:47.539 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-09-10T20:05:33.744490+0000 29'101 29:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:59:29.018775+0000 0 0 2024-09-10T20:05:47.539 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:05:47.539 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:05:47.540 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:05:47.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:47.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-10T20:05:47.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:05:47.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:05:47.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:05:47.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:05:47.563 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:05:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:05:47.604 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:47.599+0000 7f20e0a587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:05:47.606 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:47.603+0000 7f20e0a587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:05:47.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:47.603+0000 7f20e0a587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:05:48.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:05:48.048 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:05:48.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:05:48.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:05:48.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:05:48.623 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:48.619+0000 7f20e0a587c0 -1 Falling back to public interface 2024-09-10T20:05:49.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:05:49.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:05:49.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:05:49.534 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:05:49.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:05:49.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:05:49.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:05:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:50.635+0000 7f20e0a587c0 -1 osd.0 29 log_to_monitors true 2024-09-10T20:05:50.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:05:50.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:05:50.989 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:05:50.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:05:50.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:05:50.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:05:51.458 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1217649039,v1:127.0.0.1:6803/1217649039] [v2:127.0.0.1:6804/1217649039,v1:127.0.0.1:6805/1217649039] exists,up 497c7066-97a5-4ff8-bbc5-fa078793cc71 2024-09-10T20:05:51.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:05:51.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:05:51.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:05:51.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:05:51.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:05:51.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:05:51.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:05:51.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:05:51.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:05:51.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:05:51.464 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:05:51.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:05:51.498 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:51.495+0000 7f7f625617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:05:51.500 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:51.495+0000 7f7f625617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:05:51.501 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:51.499+0000 7f7f625617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:05:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:05:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:05:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:05:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:05:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:05:52.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:05:52.004 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:05:52.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:05:52.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:05:52.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:05:52.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:05:52.759 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:52.755+0000 7f7f625617c0 -1 Falling back to public interface 2024-09-10T20:05:53.496 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:05:53.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:05:53.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:05:53.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:05:53.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:05:53.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:05:53.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:05:54.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:05:54.719+0000 7f7f625617c0 -1 osd.2 29 log_to_monitors true 2024-09-10T20:05:54.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:05:54.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:05:54.934 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:05:54.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:05:54.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:05:54.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:05:55.419 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2453628059,v1:127.0.0.1:6811/2453628059] [v2:127.0.0.1:6812/2453628059,v1:127.0.0.1:6813/2453628059] exists,up da610e6e-6058-42ca-b517-21c1e0ba0f0a 2024-09-10T20:05:55.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:05:55.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:05:55.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:05:55.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:55.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-10T20:05:55.969 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-10T20:05:55.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-10T20:05:55.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-10T20:05:56.571 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-10T20:05:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:05:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-10T20:05:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:05:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:05:56.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:05:56.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:05:56.591 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:05:56.591 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:05:56.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:05:56.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:05:56.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:05:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:05:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:05:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:05:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:05:56.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:05:56.655 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:05:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:05:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:05:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:05:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:05:57.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:57.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:05:57.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-10T20:05:57.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-10T20:05:57.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-10T20:05:57.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:57.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:05:57.445 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:05:57.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:05:57.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:05:57.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:05:57.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:57.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:05:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-10T20:05:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-10T20:05:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-10T20:05:57.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:57.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-10T20:05:57.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:57.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:05:57.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-10T20:05:57.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:57.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-10T20:05:57.618 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822660 2024-09-10T20:05:57.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-10T20:05:57.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:05:58.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-10T20:05:58.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:05:59.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:05:59.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:05:59.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-10T20:05:59.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:59.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-10T20:05:59.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:59.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:05:59.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:59.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-10T20:05:59.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-10T20:05:59.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-10T20:05:59.601 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 163208757251 2024-09-10T20:05:59.601 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:06:00.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-10T20:06:00.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:06:00.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:00.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:00.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:06:00.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:00.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:00.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:00.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:00.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:00.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:00.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:01.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:01.136 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:01.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:01.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:01.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:01.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:06:01.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:06:01.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:06:01.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:01.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:06:01.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:01.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:01.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:01.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:01.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:01.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:01.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:01.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:02.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:02.284 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:02.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:02.285 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:02.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:02.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:02.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:02.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:02.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:02.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:02.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:02.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:02.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:03.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:03.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:03.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:06:03.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:03.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:06:03.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:03.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:03.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:03.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:03.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:03.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:03.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:03.662 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:04.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:04.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:04.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:04.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:04.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:04.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:06:05.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:05.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:06:05.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:05.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:05.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:05.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:05.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:05.750 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:06.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:06.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:06.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:06.202 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:06.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:06.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:06.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:06.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:06.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:06.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:06.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:06.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:06.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:07.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:07.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:07.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:06:07.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:07.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:06:08.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:08.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:08.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:08.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:08.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:08.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:08.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:08.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:08.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:08.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:08.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:08.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:09.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:09.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:09.210 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:09.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:09.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:09.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:06:09.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:09.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:06:11.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:11.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:11.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:11.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:11.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:11.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:11.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:11.400 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:11.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:11.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:11.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:11.869 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:12.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:12.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:12.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:12.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:12.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:12.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:12.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:12.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:12.489 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:13.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:13.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:13.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:06:13.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:13.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:06:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:16.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:16.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:16.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:16.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:16.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:16.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:16.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:16.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:16.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:16.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:16.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:17.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:17.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:17.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:17.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:06:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:17.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:06:24.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:24.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:24.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:24.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:24.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:24.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:24.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:24.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:24.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:24.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:24.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:24.818 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:25.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:25.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:25.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:06:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:06:25.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:06:33.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:06:33.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:06:33.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:06:33.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:06:33.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:06:33.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:06:33.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:06:33.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:06:33.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:06:33.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:06:33.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:06:33.746 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:06:34.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:06:34.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:06:34.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:06:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:06:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:06:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:06:34.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "5f6c7cec-ca0d-494b-b2ea-32884b95bee8", 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:06:35.522484+0000", 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:06:35.526 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.527 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:06:35.528 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:06:35.529 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:06:35.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.531 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:03:06.129789Z", 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:03:06.129789Z", 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:06:35.532 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:06:35.533 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.534 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.535 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:06:35.536 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 41, 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:03:07.036183+0000", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:05:56.809052+0000", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:05:54.927716+0000", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:05:56.353390+0000", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:06:35.537 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:06:35.538 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:04:54.830003+0000", 2024-09-10T20:06:35.539 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:06:35.540 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:06:35.541 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T20:06:35.542 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:06:35.543 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:06:35.544 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:06:35.544 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:06:35.544 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:06:35.544 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:06:35.544 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:06:35.544 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:06:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "score_type": "Fair distribution", 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 2.9900000095367432, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 2.9900000095367432, 2024-09-10T20:06:35.546 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 2, 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 2, 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:06:35.547 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "497c7066-97a5-4ff8-bbc5-fa078793cc71", 2024-09-10T20:06:35.548 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:06:35.548 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:06:35.548 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:06:35.548 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:06:35.548 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-10T20:06:35.548 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 36, 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 40, 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:06:35.550 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.551 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.552 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1217649039 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1217649039", 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1217649039", 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1217649039", 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1217649039", 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:06:35.554 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "cdd2bd3f-14b2-48a4-bc4b-6987c79dd657", 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:06:35.555 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 23, 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.556 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.557 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.558 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:06:35.559 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 4242344907 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/4242344907", 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/4242344907", 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4242344907", 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4242344907", 2024-09-10T20:06:35.560 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "da610e6e-6058-42ca-b517-21c1e0ba0f0a", 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 38, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-10T20:06:35.561 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.562 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.563 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.564 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2453628059 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.565 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2453628059", 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2453628059", 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2453628059", 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2453628059", 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:06:35.566 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:05:44.546147+0000", 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:03:40.508598+0000", 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:05:45.175906+0000", 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:06:35.567 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:04:16.241078+0000", 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:05:45.750653+0000", 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:06:35.568 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:04:51.381515+0000", 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:06:35.569 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:06:35.570 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1217649039,v1:127.0.0.1:6805/1217649039]", 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:06:35.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:06:35.572 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:03:13.824804Z", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:06:35.573 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1217649039,v1:127.0.0.1:6803/1217649039]", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1217649039,v1:127.0.0.1:6809/1217649039]", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1217649039,v1:127.0.0.1:6807/1217649039]", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:06:35.574 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/4242344907,v1:127.0.0.1:6813/4242344907]", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:06:35.575 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:06:35.576 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:03:48.109111Z", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:06:35.577 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/4242344907,v1:127.0.0.1:6811/4242344907]", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4242344907,v1:127.0.0.1:6817/4242344907]", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4242344907,v1:127.0.0.1:6815/4242344907]", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:06:35.578 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:06:35.579 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2453628059,v1:127.0.0.1:6813/2453628059]", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:06:35.580 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:04:23.984640Z", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:06:35.581 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2453628059,v1:127.0.0.1:6811/2453628059]", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2453628059,v1:127.0.0.1:6817/2453628059]", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2453628059,v1:127.0.0.1:6815/2453628059]", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:06:35.582 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-10T20:06:35.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 36 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 38 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:06:35.584 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 41, 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.589 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:06:35.590 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.591 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.592 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:06:35.593 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:06:35.594 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:06:35.595 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:06:35.596 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:06:35.597 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.598 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:06:35.599 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:06:35.600 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:03:07:012105+0000", 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:06:35.601 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:06:35.602 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-10T20:06:35.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:06:35.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:06:35.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:06:35.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:06:35.607 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:06:35.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83660, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2112, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489140, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036879360, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:06:35.609 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2162688, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460116, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10397, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416931 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 1, 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:06:35.610 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:06:35.611 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83660, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2112, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489140, 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:06:35.612 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036879360, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2162688, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460116, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10397, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416931 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:06:35.613 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 1, 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:06:35.614 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.616 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:06:35.617 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:06:35.618 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:06:35.619 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1228800, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 185100, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:06:35.620 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:06:35.621 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 154618822667 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672987 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 163208757259 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.622 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:06:35.624 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:05:52.974327+0000", 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "services": { 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": { 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "daemons": { 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": "", 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "0": { 2024-09-10T20:06:35.625 INFO:tasks.workunit.client.0.smithi039.stdout: "start_epoch": 0, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "start_stamp": "0.000000", 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "gid": 0, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "metadata": {}, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "task_status": {} 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 93, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:06:35.626 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 384, 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 525, 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:06:35.627 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:06:35.628 INFO:tasks.workunit.client.0.smithi039.stderr:report 2085713939 2024-09-10T20:06:35.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:06:35.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-10T20:06:35.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-10T20:06:35.628 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-10T20:06:35.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-10T20:06:35.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:35.966 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:35.967 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-10T20:05:57.086799+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:21:50.950857+0000 0 0 2024-09-10T20:06:35.967 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:35.967 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:35.967 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:35.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-10T20:06:36.113 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-10T20:06:36.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-10T20:06:36.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:36.536 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:36.536 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-10T20:05:57.086799+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:21:50.950857+0000 0 0 2024-09-10T20:06:36.536 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:36.536 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:36.536 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:36.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-10T20:06:36.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-10T20:06:36.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:06:36.553 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:06:36.553 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:06:36.553 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:06:36.553 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:06:36.554 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:06:36.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-10T20:06:36.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok set_recovery_delay 100000 2024-09-10T20:06:36.661 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:06:36.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-10T20:06:36.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-10T20:06:36.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:06:36.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:06:36.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:06:36.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:06:36.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:06:36.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:06:36.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-10T20:06:36.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok set_recovery_delay 100000 2024-09-10T20:06:36.793 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:06:36.805 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-10T20:06:36.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-10T20:06:36.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:37.229 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:37.229 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-10T20:05:57.086799+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:21:50.950857+0000 0 0 2024-09-10T20:06:37.229 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:37.229 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:37.230 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:37.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-10T20:06:37.945 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-10T20:06:37.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-10T20:06:37.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:06:37.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:06:37.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:06:37.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:06:37.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:06:37.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:06:37.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:06:37.975 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:06:37.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:06:37.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:06:37.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:06:37.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:06:37.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:06:37.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:06:37.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:06:38.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:38.010+0000 7f1cef50c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:06:38.017 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:38.014+0000 7f1cef50c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:06:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:38.014+0000 7f1cef50c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:06:38.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-10T20:06:38.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-10T20:06:39.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:39.262+0000 7f1cef50c7c0 -1 Falling back to public interface 2024-09-10T20:06:41.394 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:41.390+0000 7f1cef50c7c0 -1 osd.1 29 log_to_monitors true 2024-09-10T20:06:43.460 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-10T20:06:43.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-10T20:06:43.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:43.890 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:43.891 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-10T20:05:57.086799+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:21:50.950857+0000 0 0 2024-09-10T20:06:43.891 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:43.891 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:43.891 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:43.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-10T20:06:43.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:06:43.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:06:43.908 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:06:43.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:06:43.909 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:06:43.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:06:43.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-10T20:06:43.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok set_recovery_delay 100000 2024-09-10T20:06:44.023 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:06:44.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-10T20:06:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-10T20:06:44.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-10T20:06:44.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:06:44.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:06:44.714 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:06:44.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:06:44.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:06:45.401 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1217649039,v1:127.0.0.1:6803/1217649039] [v2:127.0.0.1:6804/1217649039,v1:127.0.0.1:6805/1217649039] exists,up 497c7066-97a5-4ff8-bbc5-fa078793cc71 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:06:45.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:06:45.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:06:45.479 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:06:45.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:06:45.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:06:45.902 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/898431583,v1:127.0.0.1:6819/898431583] [v2:127.0.0.1:6820/898431583,v1:127.0.0.1:6821/898431583] exists,up cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 2024-09-10T20:06:45.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:06:45.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:06:45.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:06:45.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-10T20:06:45.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:06:45.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:06:45.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:06:45.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:06:45.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:06:45.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:06:45.907 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:06:45.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:06:45.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:06:46.361 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2453628059,v1:127.0.0.1:6811/2453628059] [v2:127.0.0.1:6812/2453628059,v1:127.0.0.1:6813/2453628059] exists,up da610e6e-6058-42ca-b517-21c1e0ba0f0a 2024-09-10T20:06:46.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:06:46.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:06:46.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:06:46.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:46.802 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:46.802 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-10T20:05:57.086799+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:21:50.950857+0000 0 0 2024-09-10T20:06:46.802 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:46.802 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:46.802 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:46.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-10T20:06:46.819 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-10T20:06:46.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:47.241 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:47.241 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-09-10T20:06:45.331015+0000 41'101 47:140 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:06:47.241 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:47.241 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:47.241 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:47.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-10T20:06:47.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-10T20:06:47.429 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-10T20:06:47.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:47.868 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:47.869 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-09-10T20:06:45.331015+0000 41'101 47:140 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:06:47.869 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:47.869 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:47.869 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:47.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-10T20:06:47.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:06:47.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:06:47.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:06:47.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:06:47.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:06:47.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:06:47.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-10T20:06:47.994 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-10T20:06:47.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-10T20:06:48.427 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:06:48.427 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 2 2 0 0 62050 0 0 17 84 17 active+recovering+degraded 2024-09-10T20:06:45.331015+0000 41'101 47:140 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:06:48.427 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:06:48.427 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:06:48.428 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:06:48.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:06:48.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:06:48.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:06:48.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:06:48.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:06:48.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:06:48.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:06:48.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:06:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:06:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:06:48.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:06:48.449 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:06:48.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:06:48.481 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:48.478+0000 7f67784307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:06:48.483 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:48.478+0000 7f67784307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:06:48.484 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:48.482+0000 7f67784307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:06:48.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:06:48.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:06:48.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:06:48.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:06:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:06:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:06:48.929 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:06:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:06:48.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:06:48.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:06:49.450 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/898431583,v1:127.0.0.1:6819/898431583] [v2:127.0.0.1:6820/898431583,v1:127.0.0.1:6821/898431583] exists,up cdd2bd3f-14b2-48a4-bc4b-6987c79dd657 2024-09-10T20:06:49.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:06:49.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:06:49.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:06:49.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-10T20:06:49.547 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:49.542+0000 7f67784307c0 -1 Falling back to public interface 2024-09-10T20:06:51.506 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:06:51.502+0000 7f67784307c0 -1 osd.1 47 log_to_monitors true 2024-09-10T20:07:09.452 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-10T20:07:09.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-10T20:07:09.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-10T20:07:09.886 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:07:09.886 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-10T20:06:53.169535+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:24:49.714009+0000 0 0 2024-09-10T20:07:09.887 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:07:09.887 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:07:09.887 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:07:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-10T20:07:09.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-10T20:07:10.059 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:07:10.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-10T20:07:10.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-10T20:07:10.233 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:07:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-10T20:07:10.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-10T20:07:10.396 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:07:10.410 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-10T20:07:10.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-10T20:07:10.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-10T20:07:10.835 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:07:10.835 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-09-10T20:06:53.169535+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:04:54.856218+0000 0'0 2024-09-10T20:04:54.856218+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:24:49.714009+0000 0 0 2024-09-10T20:07:10.835 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:07:10.835 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:07:10.835 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:07:10.854 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-10T20:07:10.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-10T20:07:10.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:07:10.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-10T20:07:10.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:07:10.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-10T20:07:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:07:10.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-10T20:07:10.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:07:10.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-10T20:07:11.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-10T20:07:11.007 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-10T20:07:11.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:07:11.013 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2024-09-10T20:05:32.838298+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-10T20:07:11.013 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2024-09-10T20:05:32.838298+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-10T20:07:11.013 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4620.0:1 2024-09-10T20:05:32.838298+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:06:44.922+0000 7f1cd3496640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-10T20:07:11.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:07:11.019 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-10T20:07:11.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-10T20:07:11.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-10T20:07:11.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-10T20:07:11.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-10T20:07:11.621 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-10T20:07:11.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-10T20:07:11.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:07:11.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:07:11.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:07:11.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:07:11.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:07:11.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:07:11.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-10T20:07:11.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:07:11.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:07:11.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:07:11.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:07:11.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:07:11.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:07:11.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:07:11.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:07:11.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:07:11.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:07:11.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:07:11.957 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:07:11.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:07:11.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:07:11.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:07:11.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:07:11.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:07:11.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:07:11.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:07:11.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:07:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:07:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:07:11.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:07:11.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:11.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:11.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:07:11.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:07:11.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:07:11.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:07:11.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:07:11.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:07:11.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:07:11.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:07:11.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:07:11.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:07:11.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:07:11.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:07:11.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:07:11.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:07:11.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:07:11.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:07:11.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:07:11.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:07:11.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:11.993 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:11.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:07:11.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:07:11.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:07:11.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-10T20:07:11.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:07:11.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:11.995 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:11.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27146 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-10T20:07:11.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-10T20:07:11.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-10T20:07:11.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-10T20:07:11.999 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-10T20:07:11.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-10T20:07:11.999 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-10T20:07:12.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-10T20:07:12.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:07:12.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:07:12.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:07:12.472 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:07:12.472 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:12.472 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:12.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:07:12.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:07:12.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:07:12.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:07:12.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:07:12.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:07:12.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:07:12.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:07:12.660 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:07:12.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get fsid 2024-09-10T20:07:12.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:07:12.779 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:07:12.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:07:12.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:07:12.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:07:12.782 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get mon_host 2024-09-10T20:07:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-10T20:07:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-10T20:07:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:07:12.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:07:12.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:07:12.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-10T20:07:12.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:07:13.825 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:07:13.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:07:13.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:07:13.892 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:07:13.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:07:13.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:07:13.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:07:13.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ad01902e-ba29-4ebd-a261-28f4b18ec69b 2024-09-10T20:07:13.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ad01902e-ba29-4ebd-a261-28f4b18ec69b' 2024-09-10T20:07:13.895 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 ad01902e-ba29-4ebd-a261-28f4b18ec69b 2024-09-10T20:07:13.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:07:13.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxpuBm0li4NhAAFsVLGysFv3JW6KbYsRtAEw== 2024-09-10T20:07:13.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxpuBm0li4NhAAFsVLGysFv3JW6KbYsRtAEw=="}' 2024-09-10T20:07:13.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ad01902e-ba29-4ebd-a261-28f4b18ec69b -i td/divergent-priors/0/new.json 2024-09-10T20:07:14.229 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:07:14.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-10T20:07:14.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxpuBm0li4NhAAFsVLGysFv3JW6KbYsRtAEw== --osd-uuid ad01902e-ba29-4ebd-a261-28f4b18ec69b 2024-09-10T20:07:14.287 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:14.282+0000 7f60c428f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:14.297 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:14.294+0000 7f60c428f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:14.298 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:14.294+0000 7f60c428f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:14.299 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:14.294+0000 7f60c428f7c0 -1 bdev(0x55d057c6fc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:07:14.299 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:14.294+0000 7f60c428f7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-10T20:07:20.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-10T20:07:20.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:07:20.122 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:07:20.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:07:20.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:07:20.742 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:07:20.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:07:20.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:07:20.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:07:20.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:07:20.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:07:20.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:20.782+0000 7f52974207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:20.837 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:20.834+0000 7f52974207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:20.838 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:20.834+0000 7f52974207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:07:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:07:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:07:21.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:07:21.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:07:21.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:21.217 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:07:21.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:07:21.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:21.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:21.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:22.355 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:22.350+0000 7f52974207c0 -1 Falling back to public interface 2024-09-10T20:07:22.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:22.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:22.721 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:07:22.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:07:22.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:22.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:23.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:24.103 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:24.098+0000 7f52974207c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:07:24.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:24.222 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:07:24.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:24.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:07:24.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:24.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:24.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:07:25.786 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:07:25.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:25.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:26.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:27.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:27.259 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:07:27.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:27.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:07:27.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:27.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:27.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:28.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:28.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:28.814 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:07:28.815 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:07:28.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:28.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:29.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:30.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:30.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:30.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:07:30.375 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:07:30.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:30.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:30.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:31.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:31.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:31.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:07:31.952 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:07:31.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:31.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:32.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:33.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:33.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:33.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:07:33.440 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:07:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:33.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:34.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:07:34.959 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:07:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:34.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:35.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:36.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:36.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:36.506 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:07:36.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:07:36.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:36.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:36.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:37.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:37.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:37.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:07:37.996 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:07:37.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:37.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:38.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:39.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:39.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:39.931 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:07:39.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:07:39.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:39.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:40.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:41.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:41.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:41.870 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:07:41.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:07:41.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:41.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:42.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:43.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:43.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:43.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:07:43.654 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:07:43.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:43.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:44.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:45.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:45.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:45.752 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:07:45.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:07:45.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:45.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:46.350 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/729223556,v1:127.0.0.1:6803/729223556] [v2:127.0.0.1:6804/729223556,v1:127.0.0.1:6805/729223556] exists,up ad01902e-ba29-4ebd-a261-28f4b18ec69b 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:07:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:07:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:07:46.354 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:07:46.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:07:46.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:07:46.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:07:46.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:07:46.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:07:46.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 de4da426-533c-4fbf-8b8c-51563308614f' 2024-09-10T20:07:46.357 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:07:46.357 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:07:46.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASp+BmcL/rFhAAqy8cRFizijFMlwZbgnBo9A== 2024-09-10T20:07:46.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASp+BmcL/rFhAAqy8cRFizijFMlwZbgnBo9A=="}' 2024-09-10T20:07:46.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new de4da426-533c-4fbf-8b8c-51563308614f -i td/divergent-priors/1/new.json 2024-09-10T20:07:46.943 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:07:47.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-10T20:07:47.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASp+BmcL/rFhAAqy8cRFizijFMlwZbgnBo9A== --osd-uuid de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:07:47.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:47.002+0000 7f85a2c9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:47.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:47.006+0000 7f85a2c9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:47.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:47.006+0000 7f85a2c9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:47.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:47.010+0000 7f85a2c9b7c0 -1 bdev(0x55b8e6483c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:07:47.041 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:47.010+0000 7f85a2c9b7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-10T20:07:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-10T20:07:53.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:07:53.074 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:07:53.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:07:53.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:07:53.679 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:07:53.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:07:53.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:07:53.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:07:53.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:07:53.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:07:53.722 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:53.717+0000 7fafc02ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:53.723 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:53.721+0000 7fafc02ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:53.725 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:53.721+0000 7fafc02ac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:54.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:07:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:07:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:07:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:07:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:07:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:54.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:07:54.166 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:07:54.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:54.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:07:54.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:54.749 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:54.745+0000 7fafc02ac7c0 -1 Falling back to public interface 2024-09-10T20:07:55.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:55.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:55.683 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:07:55.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:07:55.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:55.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:07:56.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:56.667 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:07:56.661+0000 7fafc02ac7c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:07:57.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:57.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:57.132 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:07:57.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:07:57.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:57.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:07:57.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:58.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:58.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:58.834 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:07:58.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:07:58.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:58.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:07:59.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:00.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:00.405 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:08:00.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:08:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:00.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:00.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:01.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:01.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:01.906 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:08:01.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:08:01.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:01.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:02.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:03.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:03.376 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:08:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:08:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:03.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:03.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:04.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:04.952 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:08:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:08:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:04.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:05.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:06.405 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:08:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:08:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:06.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:06.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:07.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:07.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:07.873 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:08:07.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:08:07.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:07.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:08.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:09.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:09.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:09.336 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:08:09.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:08:09.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:09.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:10.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:11.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:11.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:11.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:08:11.212 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:08:11.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:11.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:12.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:13.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:13.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:13.189 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:08:13.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:08:13.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:13.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:14.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:15.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:15.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:15.098 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:08:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:08:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:15.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:16.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:17.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:17.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:17.117 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:08:17.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:08:17.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:17.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:18.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:19.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:19.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:19.473 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:08:19.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:08:19.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:19.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:08:19.939 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2840794077,v1:127.0.0.1:6811/2840794077] [v2:127.0.0.1:6812/2840794077,v1:127.0.0.1:6813/2840794077] exists,up de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:08:19.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:08:19.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:08:19.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:08:19.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-10T20:08:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-10T20:08:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:08:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:08:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:08:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:08:19.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:08:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:08:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:08:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:08:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:08:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:08:19.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:08:19.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:08:19.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:08:19.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:08:19.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:08:19.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:08:19.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:08:19.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:08:19.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:08:19.947 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:08:19.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:08:19.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:08:19.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:08:19.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:08:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:08:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:08:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:08:19.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:08:19.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:08:19.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d7b1b65b-16a5-4883-87b6-5b52c559f71b 2024-09-10T20:08:19.950 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 d7b1b65b-16a5-4883-87b6-5b52c559f71b 2024-09-10T20:08:19.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d7b1b65b-16a5-4883-87b6-5b52c559f71b' 2024-09-10T20:08:19.951 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:08:19.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzp+BmcNoQOhAAyA9fAzK5vyxC51beeeCO5A== 2024-09-10T20:08:19.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzp+BmcNoQOhAAyA9fAzK5vyxC51beeeCO5A=="}' 2024-09-10T20:08:19.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d7b1b65b-16a5-4883-87b6-5b52c559f71b -i td/divergent-priors/2/new.json 2024-09-10T20:08:20.501 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:08:20.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-10T20:08:20.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzp+BmcNoQOhAAyA9fAzK5vyxC51beeeCO5A== --osd-uuid d7b1b65b-16a5-4883-87b6-5b52c559f71b 2024-09-10T20:08:20.564 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:20.561+0000 7f81317517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:08:20.566 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:20.561+0000 7f81317517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:08:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:20.561+0000 7f81317517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:08:20.567 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:20.565+0000 7f81317517c0 -1 bdev(0x55a4708d3c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:08:20.568 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:20.565+0000 7f81317517c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-10T20:08:26.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-10T20:08:26.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:08:26.428 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:08:26.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:08:26.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:08:27.070 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:08:27.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:08:27.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:08:27.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:08:27.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:08:27.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:08:27.113 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:27.109+0000 7f131dc4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:08:27.127 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:27.121+0000 7f131dc4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:08:27.128 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:27.125+0000 7f131dc4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:27.559 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:08:27.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:08:27.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:27.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:28.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:28.362 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:28.357+0000 7f131dc4f7c0 -1 Falling back to public interface 2024-09-10T20:08:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:29.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:29.043 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:08:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:08:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:29.044 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:29.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:30.149 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:08:30.145+0000 7f131dc4f7c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:08:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:30.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:30.554 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:08:30.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:08:30.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:30.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:31.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:32.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:32.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:32.251 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:08:32.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:08:32.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:32.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:32.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:33.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:33.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:33.716 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:08:33.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:08:33.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:33.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:34.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:35.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:35.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:35.250 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:08:35.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:08:35.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:35.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:35.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:36.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:36.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:36.887 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:08:36.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:08:36.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:36.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:37.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:38.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:38.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:38.479 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:08:38.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:08:38.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:38.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:39.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:40.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:40.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:40.017 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:08:40.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:08:40.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:40.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:40.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:41.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:41.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:41.509 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:08:41.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:08:41.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:41.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:42.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:43.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:43.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:43.316 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:08:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:08:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:43.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:44.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:45.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:45.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:45.466 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:08:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:08:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:45.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:46.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:47.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:47.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:47.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:08:47.467 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:08:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:47.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:48.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:49.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:49.118 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:08:49.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:49.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:08:49.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:49.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:50.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:51.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:51.128 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:08:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:08:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:51.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:52.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:08:53.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:08:53.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:08:53.340 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:08:53.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:08:53.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:08:53.340 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:08:53.874 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/274394917,v1:127.0.0.1:6819/274394917] [v2:127.0.0.1:6820/274394917,v1:127.0.0.1:6821/274394917] exists,up d7b1b65b-16a5-4883-87b6-5b52c559f71b 2024-09-10T20:08:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:08:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:08:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:08:53.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-10T20:08:54.530 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-10T20:08:54.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-10T20:08:55.183 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-10T20:08:55.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-10T20:08:56.045 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-10T20:08:56.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-10T20:08:56.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T20:08:56.911 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-10T20:08:56.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:08:57.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-10T20:08:58.656 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-10T20:08:58.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-10T20:08:59.503 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-10T20:08:59.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-10T20:08:59.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:08:59.524 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:09:00.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:09:00.021 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:09:00.021 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:09:00.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:09:00.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:00.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:09:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-10T20:09:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-10T20:09:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-10T20:09:00.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:00.196 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:09:00.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672975 2024-09-10T20:09:00.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672975 2024-09-10T20:09:00.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672975' 2024-09-10T20:09:00.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:00.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:09:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509449 2024-09-10T20:09:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509449 2024-09-10T20:09:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672975 2-64424509449' 2024-09-10T20:09:00.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:00.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-10T20:09:00.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:00.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:09:00.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-10T20:09:00.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-10T20:09:00.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-10T20:09:00.544 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836503 2024-09-10T20:09:00.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:09:00.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-10T20:09:00.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:00.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672975 2024-09-10T20:09:00.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:00.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:09:00.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672975 2024-09-10T20:09:00.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:00.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672975 2024-09-10T20:09:00.990 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672975 2024-09-10T20:09:00.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-10T20:09:00.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:09:01.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-10T20:09:01.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:01.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509449 2024-09-10T20:09:01.453 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:01.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:09:01.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509449 2024-09-10T20:09:01.456 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:01.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509449 2024-09-10T20:09:01.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-10T20:09:01.457 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509449 2024-09-10T20:09:01.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-10T20:09:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-10T20:09:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:09:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:09:01.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:09:01.948 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:09:01.949 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:09:01.949 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:09:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:09:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:09:01.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:09:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:09:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:09:02.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:09:02.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:09:02.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:09:02.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:09:02.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:09:02.530 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:09:02.530 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:09:02.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:09:02.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:02.530 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:09:02.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836506 2024-09-10T20:09:02.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836506 2024-09-10T20:09:02.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506' 2024-09-10T20:09:02.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:02.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:09:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-10T20:09:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-10T20:09:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672977' 2024-09-10T20:09:02.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:02.851 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:09:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-10T20:09:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-10T20:09:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836506 1-42949672977 2-64424509451' 2024-09-10T20:09:03.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:03.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836506 2024-09-10T20:09:03.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:03.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:09:03.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836506 2024-09-10T20:09:03.009 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:03.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836506 2024-09-10T20:09:03.010 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836506 2024-09-10T20:09:03.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-10T20:09:03.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:09:03.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-10T20:09:03.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:03.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-10T20:09:03.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:03.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:09:03.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-10T20:09:03.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:03.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-10T20:09:03.487 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672977 2024-09-10T20:09:03.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-10T20:09:03.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:09:03.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-10T20:09:03.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:03.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-10T20:09:03.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:03.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:09:03.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:03.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-10T20:09:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-10T20:09:03.987 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509451 2024-09-10T20:09:03.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-10T20:09:03.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:04.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509451 2024-09-10T20:09:04.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:09:05.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:09:05.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:05.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509451 2024-09-10T20:09:05.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:09:05.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:05.961 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:09:06.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:09:06.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:09:06.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:09:06.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:09:06.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:09:06.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:09:06.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:09:06.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:09:07.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:09:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:09:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:09:07.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:07.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:09:07.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:09:07.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:09:07.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-10T20:09:07.594 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-10T20:09:08.011 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:09:08.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-10T20:09:08.028 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-10T20:09:08.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-10T20:09:08.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:09:08.465 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:09:08.465 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:08:58.038575+0000 0'0 29:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:25:44.327285+0000 0 0 2024-09-10T20:09:08.465 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:09:08.466 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:09:08.466 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:09:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-10T20:09:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-10T20:09:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-10T20:09:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-10T20:09:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-10T20:09:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-10T20:09:08.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-10T20:09:08.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-10T20:09:08.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-10T20:09:08.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-10T20:09:08.483 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-10T20:09:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-10T20:09:08.484 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-10T20:09:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-10T20:09:08.625 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:08.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-10T20:09:08.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:08.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-10T20:09:08.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:08.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-10T20:09:09.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-10T20:09:09.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-10T20:09:09.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-10T20:09:09.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-10T20:09:09.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-10T20:09:09.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-10T20:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-10T20:09:10.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:10.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-10T20:09:10.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:10.285 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-10T20:09:10.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:10.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-10T20:09:10.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:10.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-10T20:09:10.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:10.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-10T20:09:10.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:10.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-10T20:09:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-10T20:09:11.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-10T20:09:11.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-10T20:09:11.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-10T20:09:11.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-10T20:09:11.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-10T20:09:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:11.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-10T20:09:12.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:12.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-10T20:09:12.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:12.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-10T20:09:12.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:12.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-10T20:09:12.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:12.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-10T20:09:12.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:12.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-10T20:09:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:12.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-10T20:09:13.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-10T20:09:13.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-10T20:09:13.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-10T20:09:13.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-10T20:09:13.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-10T20:09:13.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-10T20:09:13.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:13.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-10T20:09:14.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-10T20:09:14.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-10T20:09:14.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-10T20:09:14.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-10T20:09:14.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-10T20:09:14.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-10T20:09:14.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:14.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-10T20:09:15.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:15.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-10T20:09:15.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:15.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-10T20:09:15.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:15.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-10T20:09:15.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:15.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-10T20:09:15.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:15.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-10T20:09:15.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:15.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-10T20:09:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-10T20:09:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-10T20:09:16.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-10T20:09:16.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-10T20:09:16.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-10T20:09:16.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-10T20:09:16.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:16.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-10T20:09:17.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:17.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-10T20:09:17.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:17.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-10T20:09:17.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:17.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-10T20:09:17.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:17.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-10T20:09:17.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:17.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-10T20:09:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-10T20:09:18.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-10T20:09:18.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-10T20:09:18.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-10T20:09:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-10T20:09:18.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-10T20:09:18.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:18.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-10T20:09:19.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:19.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-10T20:09:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:19.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-10T20:09:19.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:19.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-10T20:09:19.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:19.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-10T20:09:19.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:19.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-10T20:09:19.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:19.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-10T20:09:19.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-10T20:09:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-10T20:09:20.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-10T20:09:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-10T20:09:20.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-10T20:09:20.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-10T20:09:20.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:20.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-10T20:09:21.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-10T20:09:21.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-10T20:09:21.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-10T20:09:21.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-10T20:09:21.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-10T20:09:21.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-10T20:09:21.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:21.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-10T20:09:22.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:22.106 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-10T20:09:22.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:22.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-10T20:09:22.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:22.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-10T20:09:22.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:22.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-10T20:09:22.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:22.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-10T20:09:22.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:22.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-10T20:09:23.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:23.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-10T20:09:23.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:23.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-10T20:09:23.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:23.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-10T20:09:23.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:23.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-10T20:09:23.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-10T20:09:23.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-10T20:09:23.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:09:23.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-10T20:09:23.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:09:23.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:09:23.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:09:23.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:09:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:09:23.793 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:09:23.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:09:23.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:09:23.794 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:09:23.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:09:23.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:09:23.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:09:23.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:09:23.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:09:23.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:09:24.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:09:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:09:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:09:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:09:24.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:24.366 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:09:24.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836512 2024-09-10T20:09:24.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836512 2024-09-10T20:09:24.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512' 2024-09-10T20:09:24.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:24.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:09:24.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672983 2024-09-10T20:09:24.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672983 2024-09-10T20:09:24.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983' 2024-09-10T20:09:24.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:24.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:09:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509458 2024-09-10T20:09:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509458 2024-09-10T20:09:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836512 1-42949672983 2-64424509458' 2024-09-10T20:09:24.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:24.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836512 2024-09-10T20:09:24.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:24.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:09:24.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836512 2024-09-10T20:09:24.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:24.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836512 2024-09-10T20:09:24.892 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836512 2024-09-10T20:09:24.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-09-10T20:09:24.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:09:25.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836512 -lt 21474836512 2024-09-10T20:09:25.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:25.417 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:25.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672983 2024-09-10T20:09:25.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:09:25.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672983 2024-09-10T20:09:25.419 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:25.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672983 2024-09-10T20:09:25.421 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672983 2024-09-10T20:09:25.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-10T20:09:25.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:09:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-10T20:09:25.902 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:25.902 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509458 2024-09-10T20:09:25.903 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:25.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:09:25.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509458 2024-09-10T20:09:25.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:25.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509458 2024-09-10T20:09:25.907 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509458 2024-09-10T20:09:25.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-10T20:09:25.907 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509456 -lt 64424509458 2024-09-10T20:09:26.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:09:27.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:09:27.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:27.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-10T20:09:27.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:09:27.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:09:27.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:28.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:09:28.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:09:28.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:09:28.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:09:28.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:09:28.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:09:28.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:09:28.536 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:09:28.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:09:28.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:09:28.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:09:28.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:29.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:09:29.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:09:29.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:09:29.541 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-10T20:09:29.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-10T20:09:29.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-10T20:09:29.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-10T20:09:29.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:09:30.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-10T20:09:30.017 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-10T20:09:30.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-10T20:09:30.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:09:30.460 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:09:30.460 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-10T20:08:58.038575+0000 29'100 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:25:44.327285+0000 0 0 2024-09-10T20:09:30.460 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:09:30.460 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:09:30.461 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:09:30.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:30.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-10T20:09:30.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:09:30.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:09:30.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:09:30.477 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:09:30.478 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:09:30.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:09:30.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-10T20:09:30.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-10T20:09:30.588 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:09:30.588 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:09:30.588 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:09:30.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:30.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-10T20:09:30.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:09:30.602 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:09:30.603 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:09:30.603 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:09:30.603 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:09:30.603 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:09:30.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-10T20:09:30.604 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-10T20:09:30.709 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:09:30.709 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:09:30.709 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:09:30.722 INFO:tasks.workunit.client.0.smithi039.stdout:create new divergent objects 2024-09-10T20:09:30.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-10T20:09:30.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-10T20:09:30.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:09:30.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-10T20:09:30.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:09:30.724 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent objects 2024-09-10T20:09:30.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-10T20:09:30.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-10T20:09:30.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-10T20:09:30.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-10T20:09:30.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-10T20:09:30.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-10T20:09:30.727 INFO:tasks.workunit.client.0.smithi039.stdout:remove divergent objects 2024-09-10T20:09:30.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-10T20:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-10T20:09:30.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-10T20:09:30.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-10T20:09:30.730 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-10T20:09:30.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-10T20:09:30.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-10T20:09:30.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-10T20:09:30.732 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-10T20:09:30.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-10T20:09:30.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-10T20:09:30.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-10T20:09:40.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-10T20:09:40.741 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-10T20:09:40.742 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-10T20:09:40.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-10T20:09:40.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:09:41.191 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:09:41.191 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-09-10T20:08:58.038575+0000 29'106 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:25:44.327285+0000 0 0 2024-09-10T20:09:41.191 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:09:41.191 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:09:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:09:41.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-10T20:09:41.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:09:41.209 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:09:41.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:09:41.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:09:41.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:09:41.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:09:41.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-10T20:09:41.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-10T20:09:41.898 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:09:41.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-10T20:09:41.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-10T20:09:42.473 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:09:42.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-10T20:09:42.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-10T20:09:43.058 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:09:43.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:43.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-10T20:09:43.634 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-10T20:09:43.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:43.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-10T20:09:44.228 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-10T20:09:44.244 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-10T20:09:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-10T20:09:44.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:09:44.672 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:09:44.673 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-09-10T20:08:58.038575+0000 29'106 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:25:44.327285+0000 0 0 2024-09-10T20:09:44.673 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:09:44.673 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:09:44.673 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:09:44.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:09:44.693 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:09:44.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:09:44.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:09:44.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:09:44.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:09:44.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:09:44.695 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:09:44.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:09:44.697 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-10T20:09:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:09:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:09:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:09:44.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:09:44.726 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:44.720+0000 7f99271fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:09:44.728 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:44.724+0000 7f99271fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:09:44.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:44.724+0000 7f99271fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:09:45.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:09:45.179 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:09:45.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:09:45.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:09:45.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:09:45.994 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:45.988+0000 7f99271fc7c0 -1 Falling back to public interface 2024-09-10T20:09:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:09:46.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:09:46.667 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:09:46.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:09:46.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:09:46.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:09:47.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:09:47.961 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:47.956+0000 7f99271fc7c0 -1 osd.0 29 log_to_monitors true 2024-09-10T20:09:48.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:09:48.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:09:48.133 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:09:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:09:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:09:48.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:09:48.658 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/469311080,v1:127.0.0.1:6803/469311080] [v2:127.0.0.1:6804/469311080,v1:127.0.0.1:6805/469311080] exists,up ad01902e-ba29-4ebd-a261-28f4b18ec69b 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:09:48.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:09:48.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:09:48.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:09:48.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:09:48.666 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:09:48.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:09:48.666 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-10T20:09:48.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:09:48.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:09:48.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:09:48.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:09:48.703 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:48.700+0000 7f8702fe87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:09:48.720 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:48.716+0000 7f8702fe87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:09:48.721 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:48.716+0000 7f8702fe87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:09:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:09:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:09:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:09:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:09:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:09:49.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:09:49.152 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:09:49.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:09:49.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:09:49.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:09:49.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:09:50.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:50.004+0000 7f8702fe87c0 -1 Falling back to public interface 2024-09-10T20:09:50.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:09:50.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:09:50.637 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:09:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:09:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:09:50.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:09:51.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:09:51.948 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:09:51.944+0000 7f8702fe87c0 -1 osd.2 29 log_to_monitors true 2024-09-10T20:09:52.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:09:52.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:09:52.104 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:09:52.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:09:52.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:09:52.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:09:52.592 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2440756695,v1:127.0.0.1:6811/2440756695] [v2:127.0.0.1:6812/2440756695,v1:127.0.0.1:6813/2440756695] exists,up d7b1b65b-16a5-4883-87b6-5b52c559f71b 2024-09-10T20:09:52.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:09:52.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:09:52.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:09:52.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:52.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-10T20:09:53.250 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-10T20:09:53.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:09:53.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-10T20:09:53.818 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-10T20:09:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:09:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-10T20:09:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:09:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:09:53.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:09:53.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:09:53.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:09:53.836 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:09:53.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:09:53.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:09:53.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:09:53.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:09:53.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:09:53.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:09:53.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:09:53.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:09:53.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:09:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:09:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:09:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:09:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:09:54.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:54.478 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:09:54.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822660 2024-09-10T20:09:54.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822660 2024-09-10T20:09:54.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660' 2024-09-10T20:09:54.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:54.652 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:09:54.778 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:09:54.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:09:54.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:09:54.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:09:54.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:09:54.780 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:09:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757251 2024-09-10T20:09:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757251 2024-09-10T20:09:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-10T20:09:55.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:55.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822660 2024-09-10T20:09:55.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:55.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:09:55.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822660 2024-09-10T20:09:55.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:55.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822660 2024-09-10T20:09:55.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-10T20:09:55.018 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822660 2024-09-10T20:09:55.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:09:55.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-10T20:09:55.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:09:55.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757251 2024-09-10T20:09:55.518 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:09:55.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:09:55.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757251 2024-09-10T20:09:55.519 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:09:55.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757251 2024-09-10T20:09:55.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-10T20:09:55.521 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 163208757251 2024-09-10T20:09:55.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:56.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 163208757251 2024-09-10T20:09:56.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:09:57.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:09:57.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:09:57.495 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-10T20:09:57.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:09:57.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:09:57.495 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:58.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:09:58.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:09:58.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:09:58.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:09:58.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:09:58.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:09:58.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:09:58.113 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:09:58.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:09:58.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:09:58.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:09:58.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:09:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:09:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:09:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:09:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:09:59.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:09:59.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:09:59.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:09:59.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:09:59.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:09:59.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:09:59.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:09:59.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:09:59.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:09:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:09:59.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:09:59.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:09:59.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:00.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:00.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:00.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:00.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:00.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:00.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:00.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:00.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:00.282 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:10:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:00.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:10:01.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:01.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:01.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:01.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:01.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:01.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:01.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:01.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:01.515 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:01.515 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:01.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:01.516 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:02.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:02.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:02.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:02.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:02.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:02.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:02.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:02.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:02.110 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:02.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:02.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:02.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:10:02.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:02.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:10:03.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:03.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:03.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:03.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:03.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:03.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:03.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:03.099 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:03.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:03.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:03.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:03.542 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:04.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:04.129 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:10:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:10:05.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:05.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:05.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:05.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:05.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:05.526 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:05.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:05.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:05.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:05.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:05.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:05.974 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:06.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:06.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:06.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:06.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:06.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:06.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:06.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:06.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:06.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:07.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:07.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:07.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:10:07.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:07.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:10:08.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:08.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:08.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:08.738 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:09.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:09.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:09.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:09.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:09.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:09.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:09.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:09.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:09.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:09.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:09.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:09.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:09.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:10.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:10.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:10.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:10:10.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:10.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:10:13.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:13.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:13.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:13.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:13.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:13.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:13.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:13.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:14.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:14.014 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:14.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:14.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:14.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:14.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:14.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:14.630 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:10:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:15.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:10:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:21.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:21.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:21.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:21.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:21.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:21.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:21.657 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:22.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:22.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:22.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:22.108 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:22.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:22.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:22.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:23.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:10:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:23.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:10:30.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:30.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:30.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:30.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:30.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:30.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:30.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:30.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:31.055 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:31.055 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:31.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:31.056 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:31.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:31.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:32.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:32.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:32.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:10:32.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:10:32.813 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:10:32.813 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "40a5cac5-3fe2-4b72-8a15-1f9550647f0b", 2024-09-10T20:10:32.813 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:10:32.813 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:10:32.813 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:10:32.812293+0000", 2024-09-10T20:10:32.813 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.814 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.815 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.816 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:10:32.817 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:07:12.025972Z", 2024-09-10T20:10:32.818 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:07:12.025972Z", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:10:32.819 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:10:32.820 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:10:32.821 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.822 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 41, 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:07:12.940975+0000", 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:09:54.118480+0000", 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:09:52.228289+0000", 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:09:53.650477+0000", 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:10:32.823 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:10:32.824 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:10:32.824 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:10:32.824 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:10:32.824 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:10:32.824 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:10:32.825 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:10:32.825 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:10:32.825 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.825 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-10T20:10:32.825 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:10:32.825 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:10:32.826 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:10:32.827 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:10:32.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.827 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:10:32.827 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:10:32.827 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:08:56.443695+0000", 2024-09-10T20:10:32.827 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:10:32.828 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:10:32.829 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-10T20:10:32.830 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:10:32.831 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:10:32.832 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "score_type": "Fair distribution", 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 2.9900000095367432, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 2.9900000095367432, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 2, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 2, 2024-09-10T20:10:32.833 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "ad01902e-ba29-4ebd-a261-28f4b18ec69b", 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:10:32.834 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 36, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 40, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.835 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:10:32.836 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.837 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:10:32.838 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:10:32.839 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.840 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.840 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.840 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/469311080", 2024-09-10T20:10:32.840 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/469311080", 2024-09-10T20:10:32.840 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/469311080", 2024-09-10T20:10:32.840 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/469311080", 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "de4da426-533c-4fbf-8b8c-51563308614f", 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:10:32.841 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 23, 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.843 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:10:32.844 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:10:32.845 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2840794077", 2024-09-10T20:10:32.846 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2840794077", 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2840794077", 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2840794077", 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "d7b1b65b-16a5-4883-87b6-5b52c559f71b", 2024-09-10T20:10:32.847 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 38, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.848 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.849 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:10:32.850 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:10:32.851 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2440756695", 2024-09-10T20:10:32.852 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2440756695", 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2440756695", 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2440756695", 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:10:32.853 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:09:41.698707+0000", 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:07:45.603787+0000", 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:09:42.301674+0000", 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:10:32.854 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:08:19.342021+0000", 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:09:42.870796+0000", 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:10:32.855 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:08:52.330693+0000", 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:10:32.856 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:10:32.857 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/469311080,v1:127.0.0.1:6805/469311080]", 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:10:32.858 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:10:32.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:07:19.280933Z", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:10:32.860 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/469311080,v1:127.0.0.1:6803/469311080]", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/469311080,v1:127.0.0.1:6809/469311080]", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/469311080,v1:127.0.0.1:6807/469311080]", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:10:32.861 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2840794077,v1:127.0.0.1:6813/2840794077]", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:10:32.862 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:10:32.863 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:07:52.413363Z", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:10:32.864 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2840794077,v1:127.0.0.1:6811/2840794077]", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2840794077,v1:127.0.0.1:6817/2840794077]", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2840794077,v1:127.0.0.1:6815/2840794077]", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:10:32.865 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:10:32.866 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:10:32.866 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:10:32.866 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:10:32.866 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:10:32.866 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:10:32.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2440756695,v1:127.0.0.1:6813/2440756695]", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:10:32.868 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:10:32.869 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:08:25.762747Z", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2440756695,v1:127.0.0.1:6811/2440756695]", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2440756695,v1:127.0.0.1:6817/2440756695]", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2440756695,v1:127.0.0.1:6815/2440756695]", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:10:32.870 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.871 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 36 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:10:32.872 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 38 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 41, 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:10:32.874 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:10:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:10:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:10:32.875 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:10:32.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.877 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.878 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:10:32.879 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:10:32.880 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.881 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:10:32.882 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:10:32.883 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.884 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:10:32.885 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:10:32.886 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.887 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:10:32.888 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:10:32.889 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.890 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:10:32.890 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:10:32.890 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:07:12:916882+0000", 2024-09-10T20:10:32.890 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:10:32.890 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stderr:report 3130637416 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:10:32.891 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:10:32.892 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:10:32.893 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:10:32.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:10:32.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:10:32.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:10:32.897 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-10T20:10:32.898 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:10:32.899 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83652, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489148, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036887552, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460272, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:10:32.900 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:10:32.901 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:10:32.901 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:10:32.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.901 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:10:32.901 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:10:32.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:10:32.902 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83652, 2024-09-10T20:10:32.903 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489148, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036887552, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460272, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:10:32.904 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.905 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:10:32.906 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 61700, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:10:32.907 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 100, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 100, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:10:32.908 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:10:32.909 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 184566, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:10:32.910 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 16, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 16, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 154618822667 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.911 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.912 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:10:32.912 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672987 2024-09-10T20:10:32.912 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.912 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.912 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 163208757258 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.913 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:10:32.914 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:10:32.915 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:10:32.916 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 4, 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:09:20.848049+0000", 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "services": { 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": { 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "daemons": { 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": "", 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "2": { 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "start_epoch": 0, 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "start_stamp": "0.000000", 2024-09-10T20:10:32.917 INFO:tasks.workunit.client.0.smithi039.stdout: "gid": 0, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "(unrecognized address family 0)/0", 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "metadata": {}, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "task_status": {} 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 88, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 366, 2024-09-10T20:10:32.918 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 501, 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:10:32.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:10:32.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-10T20:10:32.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-10T20:10:32.920 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-10T20:10:32.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-10T20:10:32.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:10:33.254 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:10:33.255 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-10T20:09:54.337416+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:12:36.825399+0000 0 0 2024-09-10T20:10:33.255 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:10:33.255 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:10:33.255 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:10:33.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-10T20:10:33.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:10:33.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-10T20:10:33.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:10:33.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:10:33.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:10:33.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:10:33.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:10:33.374 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:10:33.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:10:33.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:10:33.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:10:33.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:10:33.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:10:33.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:10:33.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:10:33.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:10:33.441 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:10:33.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:10:33.933 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:10:33.933 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:10:33.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:10:33.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:10:33.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:10:34.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=154618822670 2024-09-10T20:10:34.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 154618822670 2024-09-10T20:10:34.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822670' 2024-09-10T20:10:34.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:10:34.114 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:10:34.246 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:10:34.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:10:34.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757261 2024-09-10T20:10:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757261 2024-09-10T20:10:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-09-10T20:10:34.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:10:34.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-154618822670 2024-09-10T20:10:34.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:10:34.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:10:34.412 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-154618822670 2024-09-10T20:10:34.413 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:10:34.413 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=154618822670 2024-09-10T20:10:34.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-09-10T20:10:34.414 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 154618822670 2024-09-10T20:10:34.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:10:34.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822667 -lt 154618822670 2024-09-10T20:10:34.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:10:35.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:10:35.884 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:10:36.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 154618822670 -lt 154618822670 2024-09-10T20:10:36.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:10:36.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-163208757261 2024-09-10T20:10:36.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:10:36.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:10:36.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-163208757261 2024-09-10T20:10:36.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:10:36.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757261 2024-09-10T20:10:36.382 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 163208757261 2024-09-10T20:10:36.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-09-10T20:10:36.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:10:36.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757261 -lt 163208757261 2024-09-10T20:10:36.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:10:36.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:36.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:10:37.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:37.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:37.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:37.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:37.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:37.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:37.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:37.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:37.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:37.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:37.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:38.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:38.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:10:38.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:10:38.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:10:38.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:38.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:10:38.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:38.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:38.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:38.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:38.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:38.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:38.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:38.577 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:39.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:39.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:39.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:39.030 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:39.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:39.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:39.632 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:40.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:40.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:40.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:10:40.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:40.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:10:40.443 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:40.444 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:40.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:40.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:40.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:40.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:40.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:40.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:40.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:40.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:40.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:40.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:41.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:41.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:41.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:41.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:41.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:41.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:41.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:41.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:41.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:42.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:42.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:42.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:10:42.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:42.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:10:42.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:42.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:42.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:42.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:42.908 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:42.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:42.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:43.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:43.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:43.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:43.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:43.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:44.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:44.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:44.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:10:44.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:44.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:10:44.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:44.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:44.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:44.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:44.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:44.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:44.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:44.880 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:45.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:45.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:45.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:45.353 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:45.943 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:45.944 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:45.945 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:46.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:46.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:46.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:10:46.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:46.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:10:48.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:48.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:48.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:48.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:48.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:48.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:48.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:48.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:48.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:48.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:48.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:48.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:49.153 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:49.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:49.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:49.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:49.154 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:49.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:49.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:49.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:10:49.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:49.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:10:52.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:10:52.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:10:52.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:10:52.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:10:52.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:10:52.935 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:10:52.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:10:52.936 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:10:53.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:10:53.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:10:53.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:10:53.416 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:10:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:10:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:10:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:10:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:10:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:10:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:10:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:10:54.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:10:54.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:10:54.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:10:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:10:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:10:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:10:54.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:11:00.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:11:00.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:11:00.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:11:00.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:11:00.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:11:00.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:11:00.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:11:00.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:11:01.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:11:01.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:11:01.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:11:01.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:11:02.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:11:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:11:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:11:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:11:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:11:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:11:02.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:11:02.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:11:02.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:11:02.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:11:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:11:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:11:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:11:02.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:11:09.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:11:09.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:11:09.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:11:09.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:11:09.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:11:09.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:11:09.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:11:09.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:11:10.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:11:10.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:11:10.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:11:10.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:11:10.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:11:10.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:11:10.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:11:11.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:11:11.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:11:11.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:11:11.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:11:12.161 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "40a5cac5-3fe2-4b72-8a15-1f9550647f0b", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:11:12.160965+0000", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:11:12.162 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:11:12.163 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 2 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.164 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is stuck undersized for 76s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:11:12.165 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:11:12.166 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:07:12.025972Z", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:07:12.025972Z", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:11:12.167 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:11:12.168 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:11:12.169 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 41, 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:07:12.940975+0000", 2024-09-10T20:11:12.170 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:09:54.118480+0000", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:09:52.228289+0000", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:09:53.650477+0000", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:11:12.171 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:11:12.172 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:08:56.443695+0000", 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:11:12.173 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-10T20:11:12.174 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "29", 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:11:12.175 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:11:12.176 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:11:12.177 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "score_type": "Fair distribution", 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 2.9900000095367432, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 2.9900000095367432, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-10T20:11:12.178 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 2, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 2, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "ad01902e-ba29-4ebd-a261-28f4b18ec69b", 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:11:12.179 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 36, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 40, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 30, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.180 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.181 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.182 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.183 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 469311080 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/469311080", 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/469311080", 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/469311080", 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/469311080", 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:11:12.184 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "de4da426-533c-4fbf-8b8c-51563308614f", 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:11:12.185 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 23, 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:11:12.186 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:11:12.187 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:11:12.188 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:11:12.189 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2840794077 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2840794077", 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2840794077", 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2840794077", 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2840794077", 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:11:12.190 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "d7b1b65b-16a5-4883-87b6-5b52c559f71b", 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 38, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.191 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.192 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.193 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:11:12.194 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 2440756695 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/2440756695", 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/2440756695", 2024-09-10T20:11:12.195 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2440756695", 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2440756695", 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:09:41.698707+0000", 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:11:12.196 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:07:45.603787+0000", 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:09:42.301674+0000", 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:08:19.342021+0000", 2024-09-10T20:11:12.197 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:09:42.870796+0000", 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:08:52.330693+0000", 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:11:12.198 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.199 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.200 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/469311080,v1:127.0.0.1:6805/469311080]", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:11:12.201 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:07:19.280933Z", 2024-09-10T20:11:12.202 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/469311080,v1:127.0.0.1:6803/469311080]", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/469311080,v1:127.0.0.1:6809/469311080]", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/469311080,v1:127.0.0.1:6807/469311080]", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:11:12.203 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:11:12.204 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2840794077,v1:127.0.0.1:6813/2840794077]", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:11:12.205 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:07:52.413363Z", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:11:12.206 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2840794077,v1:127.0.0.1:6811/2840794077]", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2840794077,v1:127.0.0.1:6817/2840794077]", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2840794077,v1:127.0.0.1:6815/2840794077]", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:11:12.207 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/2440756695,v1:127.0.0.1:6813/2440756695]", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:11:12.208 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:11:12.209 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:08:25.762747Z", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:11:12.210 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/2440756695,v1:127.0.0.1:6811/2440756695]", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2440756695,v1:127.0.0.1:6817/2440756695]", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2440756695,v1:127.0.0.1:6815/2440756695]", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:11:12.211 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:11:12.212 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 36 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 38 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 41, 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:11:12.213 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.214 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:11:12.215 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.216 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:11:12.217 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:11:12.218 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:11:12.219 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:11:12.220 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:11:12.221 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:11:12.222 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:11:12.223 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:11:12.224 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:11:12.225 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:11:12.226 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.227 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:11:12.228 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.228 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:11:12.228 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:11:12.228 INFO:tasks.workunit.client.0.smithi039.stderr:report 3666021260 2024-09-10T20:11:12.228 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:07:12:916882+0000", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:11:12.229 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:11:12.230 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62050, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-10T20:11:12.231 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:11:12.232 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:11:12.233 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:11:12.234 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:11:12.235 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 17, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 17, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:11:12.236 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83652, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489148, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036887552, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460972, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:11:12.237 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:11:12.238 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:11:12.239 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83652, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2104, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489148, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036887552, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:11:12.240 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2154496, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 460972, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:11:12.241 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:11:12.242 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62050, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 100, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 300, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 100, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 100, 2024-09-10T20:11:12.243 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:11:12.244 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:11:12.245 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 185266, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.246 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 17, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 17, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 2, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 2, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 154618822677 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:11:12.247 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672987 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 163208757268 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:11:12.248 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:11:12.249 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.250 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 6, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:10:52.871435+0000", 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 108, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 432, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 587, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:11:12.251 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:11:12.252 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:11:12.252 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:11:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:11:12.252 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-10T20:11:12.252 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-10T20:11:12.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:12.580 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:12.580 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-10T20:09:54.337416+0000 41'101 41:131 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:12:36.825399+0000 0 0 2024-09-10T20:11:12.580 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:12.580 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:12.580 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:12.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:11:12.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-10T20:11:12.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:11:12.597 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:11:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:11:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:11:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:11:12.598 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:11:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-10T20:11:12.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok set_recovery_delay 100000 2024-09-10T20:11:12.711 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:11:12.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-10T20:11:12.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-10T20:11:12.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:11:12.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:11:12.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:11:12.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:11:12.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:11:12.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:11:12.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-10T20:11:12.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok set_recovery_delay 100000 2024-09-10T20:11:12.838 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:11:12.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-10T20:11:12.851 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-10T20:11:12.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:13.281 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:13.281 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-10T20:09:54.337416+0000 41'101 41:131 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:12:36.825399+0000 0 0 2024-09-10T20:11:13.281 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:13.281 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:13.281 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:13.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-10T20:11:13.981 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-10T20:11:14.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-10T20:11:14.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:11:14.006 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:11:14.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:11:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:11:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:11:14.008 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:11:14.009 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:11:14.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:11:14.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:11:14.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:11:14.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:11:14.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:11:14.011 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:11:14.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:11:14.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:11:14.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:11:14.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:11:14.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:11:14.012 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:11:14.060 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:14.059+0000 7fc6825a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:11:14.062 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:14.063+0000 7fc6825a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:11:14.064 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:14.063+0000 7fc6825a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:11:14.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-10T20:11:14.539 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-10T20:11:15.305 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:15.303+0000 7fc6825a37c0 -1 Falling back to public interface 2024-09-10T20:11:17.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:17.315+0000 7fc6825a37c0 -1 osd.1 29 log_to_monitors true 2024-09-10T20:11:19.540 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-10T20:11:19.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-10T20:11:19.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:19.983 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:19.983 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-09-10T20:09:54.337416+0000 41'101 44:137 [0,2] 0 [0,2] 0 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:12:36.825399+0000 0 0 2024-09-10T20:11:19.983 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:19.984 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:19.984 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:20.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:11:20.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-10T20:11:20.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok set_recovery_delay 100000 2024-09-10T20:11:20.114 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:11:20.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-10T20:11:20.828 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-10T20:11:20.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-10T20:11:20.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:11:20.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:11:20.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:11:20.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:11:20.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:11:20.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:11:20.854 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:11:20.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:11:20.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:11:21.429 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/469311080,v1:127.0.0.1:6803/469311080] [v2:127.0.0.1:6804/469311080,v1:127.0.0.1:6805/469311080] exists,up ad01902e-ba29-4ebd-a261-28f4b18ec69b 2024-09-10T20:11:21.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:11:21.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:11:21.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:11:21.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-10T20:11:21.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:11:21.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:11:21.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:11:21.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:11:21.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:11:21.433 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:11:21.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:11:21.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:11:21.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:11:21.928 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2848573300,v1:127.0.0.1:6819/2848573300] [v2:127.0.0.1:6820/2848573300,v1:127.0.0.1:6821/2848573300] exists,up de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:11:21.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:11:21.930 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:11:21.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:11:21.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:11:21.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:11:21.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:11:22.497 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2440756695,v1:127.0.0.1:6811/2440756695] [v2:127.0.0.1:6812/2440756695,v1:127.0.0.1:6813/2440756695] exists,up d7b1b65b-16a5-4883-87b6-5b52c559f71b 2024-09-10T20:11:22.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:11:22.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:11:22.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:11:22.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:22.950 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:22.951 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 peering 2024-09-10T20:11:21.064554+0000 29'106 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:11:22.951 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:22.951 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:22.951 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:22.968 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-10T20:11:22.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-10T20:11:22.969 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:23.473 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:23.473 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 peering 2024-09-10T20:11:21.064554+0000 29'106 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:11:23.474 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:23.474 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:23.474 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:23.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-10T20:11:23.613 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-10T20:11:23.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-10T20:11:23.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:24.048 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:24.048 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 peering 2024-09-10T20:11:21.064554+0000 29'106 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:11:24.048 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:24.048 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:24.048 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:24.064 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-10T20:11:24.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:11:24.065 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:11:24.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:11:24.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:11:24.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:11:24.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:11:24.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.27146.out 2024-09-10T20:11:24.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27146.out 2024-09-10T20:11:24.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:11:24.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:11:24.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:11:24.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:11:24.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-10T20:11:24.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27146.out 2024-09-10T20:11:25.749 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-10T20:11:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-10T20:11:25.751 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-10T20:11:25.752 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-10T20:11:25.752 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-10T20:11:25.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-10T20:11:25.753 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-10T20:11:25.754 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-10T20:11:25.754 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:169157fa:::existing_81:head# 2024-09-10T20:11:25.755 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:17731382:::existing_58:head# 2024-09-10T20:11:25.755 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-10T20:11:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-10T20:11:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-10T20:11:25.756 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-10T20:11:25.757 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-10T20:11:25.757 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:28566edb:::existing_78:head# 2024-09-10T20:11:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2a101182:::existing_20:head# 2024-09-10T20:11:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-10T20:11:25.759 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-10T20:11:25.759 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-10T20:11:25.760 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-10T20:11:25.760 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-10T20:11:25.760 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-10T20:11:25.761 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-10T20:11:25.761 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-10T20:11:25.762 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-10T20:11:25.762 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-10T20:11:25.763 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:53588912:::existing_9:head# 2024-09-10T20:11:25.763 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-10T20:11:25.763 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-10T20:11:25.764 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-10T20:11:25.764 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-10T20:11:25.765 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:602f83fe:::foo:head# 2024-09-10T20:11:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-10T20:11:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:628918ca:::existing_79:head# 2024-09-10T20:11:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-10T20:11:25.766 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-10T20:11:25.767 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-10T20:11:25.767 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-10T20:11:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-10T20:11:25.768 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-10T20:11:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-10T20:11:25.769 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-10T20:11:25.770 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:750ace21:::existing_19:head# 2024-09-10T20:11:25.770 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-10T20:11:25.771 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:79881152:::existing_63:head# 2024-09-10T20:11:25.771 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-10T20:11:25.771 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:81490677:::existing_76:head# 2024-09-10T20:11:25.772 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-10T20:11:25.772 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-10T20:11:25.773 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-10T20:11:25.773 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-10T20:11:25.773 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:92731e71:::existing_89:head# 2024-09-10T20:11:25.774 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:928205cf:::existing_82:head# 2024-09-10T20:11:25.774 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:93be2267:::existing_24:head# 2024-09-10T20:11:25.775 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-10T20:11:25.775 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-10T20:11:25.775 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-10T20:11:25.776 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-10T20:11:25.777 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-10T20:11:25.777 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a504a533:::existing_90:head# 2024-09-10T20:11:25.777 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a691337e:::existing_61:head# 2024-09-10T20:11:25.778 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-10T20:11:25.778 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-10T20:11:25.778 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-10T20:11:25.779 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-10T20:11:25.779 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-10T20:11:25.780 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b343e277:::existing_88:head# 2024-09-10T20:11:25.780 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-10T20:11:25.781 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-10T20:11:25.781 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-10T20:11:25.782 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-10T20:11:25.782 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:bebec877:::existing_74:head# 2024-09-10T20:11:25.783 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-10T20:11:25.783 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c156033e:::existing_100:head# 2024-09-10T20:11:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-10T20:11:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-10T20:11:25.784 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-10T20:11:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-10T20:11:25.785 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-10T20:11:25.786 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-10T20:11:25.786 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-10T20:11:25.787 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-10T20:11:25.787 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-10T20:11:25.788 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-10T20:11:25.788 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-10T20:11:25.789 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-10T20:11:25.789 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-10T20:11:25.789 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-10T20:11:25.790 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:e255b710:::existing_55:head# 2024-09-10T20:11:25.790 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-10T20:11:25.791 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-10T20:11:25.791 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-10T20:11:25.791 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f6809456:::existing_11:head# 2024-09-10T20:11:25.792 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-10T20:11:25.792 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-10T20:11:25.793 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-10T20:11:25.793 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-10T20:11:25.793 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-10T20:11:25.794 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-10T20:11:25.794 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-10T20:11:25.794 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-10T20:11:26.007 INFO:tasks.workunit.client.0.smithi039.stderr:Remove successful 2024-09-10T20:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27146.out 2024-09-10T20:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:11:26.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:11:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:11:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-10T20:11:26.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27146.out 2024-09-10T20:11:27.563 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-09-10T20:11:27.563 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.0 2024-09-10T20:11:27.564 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-10T20:11:27.564 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.627 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-10T20:11:27.627 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.661 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-10T20:11:27.661 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.694 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-10T20:11:27.694 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.727 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-10T20:11:27.728 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.761 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-10T20:11:27.761 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.803 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-10T20:11:27.803 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.845 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:169157fa:::existing_81:head# 2024-09-10T20:11:27.845 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.878 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:17731382:::existing_58:head# 2024-09-10T20:11:27.878 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.911 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-10T20:11:27.912 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.945 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-10T20:11:27.945 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:27.978 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-10T20:11:27.979 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.012 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-10T20:11:28.012 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.045 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-10T20:11:28.045 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.078 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:28566edb:::existing_78:head# 2024-09-10T20:11:28.079 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.112 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2a101182:::existing_20:head# 2024-09-10T20:11:28.112 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.146 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-10T20:11:28.146 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.179 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-10T20:11:28.179 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.221 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-10T20:11:28.221 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.254 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-10T20:11:28.255 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.288 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-10T20:11:28.288 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.321 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-10T20:11:28.321 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.355 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-10T20:11:28.355 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.388 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-10T20:11:28.388 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.422 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-10T20:11:28.422 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.455 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-10T20:11:28.455 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.488 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:53588912:::existing_9:head# 2024-09-10T20:11:28.489 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.522 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-10T20:11:28.522 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.555 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-10T20:11:28.556 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.589 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-10T20:11:28.589 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.622 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-10T20:11:28.623 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.656 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:602f83fe:::foo:head# 2024-09-10T20:11:28.656 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.689 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-10T20:11:28.689 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.723 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:628918ca:::existing_79:head# 2024-09-10T20:11:28.723 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.756 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-10T20:11:28.756 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.790 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-10T20:11:28.790 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.823 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-10T20:11:28.823 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.856 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-10T20:11:28.857 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.890 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-10T20:11:28.890 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.923 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-10T20:11:28.924 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:28.970 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-10T20:11:28.970 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.049 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-10T20:11:29.049 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.103 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:750ace21:::existing_19:head# 2024-09-10T20:11:29.103 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.175 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-10T20:11:29.175 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.242 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:79881152:::existing_63:head# 2024-09-10T20:11:29.242 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.284 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-10T20:11:29.284 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.325 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:81490677:::existing_76:head# 2024-09-10T20:11:29.326 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.359 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-10T20:11:29.359 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.392 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-10T20:11:29.392 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.426 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-10T20:11:29.426 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.459 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-10T20:11:29.459 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.493 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:92731e71:::existing_89:head# 2024-09-10T20:11:29.493 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.529 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:928205cf:::existing_82:head# 2024-09-10T20:11:29.529 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.559 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:93be2267:::existing_24:head# 2024-09-10T20:11:29.560 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.593 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-10T20:11:29.593 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.626 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-10T20:11:29.627 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.660 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-10T20:11:29.660 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.693 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-10T20:11:29.693 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.727 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-10T20:11:29.727 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.760 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a504a533:::existing_90:head# 2024-09-10T20:11:29.760 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.793 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a691337e:::existing_61:head# 2024-09-10T20:11:29.794 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.827 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-10T20:11:29.827 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.860 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-10T20:11:29.861 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.894 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-10T20:11:29.894 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.927 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-10T20:11:29.928 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:29.994 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-10T20:11:29.995 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.028 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b343e277:::existing_88:head# 2024-09-10T20:11:30.028 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.061 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-10T20:11:30.062 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.095 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-10T20:11:30.095 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.128 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-10T20:11:30.128 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.162 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-10T20:11:30.162 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.195 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:bebec877:::existing_74:head# 2024-09-10T20:11:30.195 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.229 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-10T20:11:30.229 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.262 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c156033e:::existing_100:head# 2024-09-10T20:11:30.262 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.296 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-10T20:11:30.296 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.329 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-10T20:11:30.329 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.362 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-10T20:11:30.363 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.396 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-10T20:11:30.396 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.429 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-10T20:11:30.429 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.463 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-10T20:11:30.463 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.496 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-10T20:11:30.496 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.530 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-10T20:11:30.530 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.563 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-10T20:11:30.564 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.597 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-10T20:11:30.597 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.630 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-10T20:11:30.630 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.664 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-10T20:11:30.664 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.697 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-10T20:11:30.697 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.730 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-10T20:11:30.731 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.764 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:e255b710:::existing_55:head# 2024-09-10T20:11:30.764 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.799 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-10T20:11:30.799 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.831 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-10T20:11:30.831 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.864 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-10T20:11:30.864 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.898 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f6809456:::existing_11:head# 2024-09-10T20:11:30.898 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.931 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-10T20:11:30.931 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:30.986 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-10T20:11:30.986 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:31.045 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-10T20:11:31.045 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:31.157 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-10T20:11:31.157 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:11:31.224 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-10T20:11:31.266 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-10T20:11:31.806 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-10T20:11:31.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-10T20:11:31.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:32.245 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:32.245 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 peering 2024-09-10T20:11:21.064554+0000 29'106 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:11:32.245 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:32.245 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:32.246 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:11:32.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:11:32.265 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:11:32.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:11:32.268 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:11:32.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:11:32.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:11:32.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:11:32.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:11:32.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:11:32.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:11:32.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:32.307+0000 7fa6896ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:11:32.313 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:32.311+0000 7fa6896ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:11:32.313 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:32.311+0000 7fa6896ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:11:32.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:11:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:11:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:11:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:11:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:11:32.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:11:32.731 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:11:32.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:11:32.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:11:32.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:11:33.185 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2848573300,v1:127.0.0.1:6819/2848573300] [v2:127.0.0.1:6820/2848573300,v1:127.0.0.1:6821/2848573300] exists,up de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:11:33.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:11:33.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:11:33.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:11:33.187 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:11:33.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:11:33.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:11:33.380 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:33.379+0000 7fa6896ca7c0 -1 Falling back to public interface 2024-09-10T20:11:33.673 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2848573300,v1:127.0.0.1:6819/2848573300] [v2:127.0.0.1:6820/2848573300,v1:127.0.0.1:6821/2848573300] exists,up de4da426-533c-4fbf-8b8c-51563308614f 2024-09-10T20:11:33.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:11:33.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:11:33.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:11:33.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-10T20:11:35.356 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:11:35.355+0000 7fa6896ca7c0 -1 osd.1 47 log_to_monitors true 2024-09-10T20:11:53.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-10T20:11:53.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:11:53.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:11:53.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:11:53.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:11:53.677 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:11:53.678 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:11:53.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-10T20:11:53.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok dump_ops_in_flight 2024-09-10T20:11:53.795 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:11:53.795 INFO:tasks.workunit.client.0.smithi039.stdout: "ops": [], 2024-09-10T20:11:53.795 INFO:tasks.workunit.client.0.smithi039.stdout: "num_ops": 0 2024-09-10T20:11:53.795 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:11:53.807 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-10T20:11:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-10T20:11:53.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:54.240 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:54.240 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-10T20:11:37.461794+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:06:37.671994+0000 0 0 2024-09-10T20:11:54.241 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:54.241 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:54.241 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:54.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-10T20:11:54.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-10T20:11:54.414 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:11:54.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-10T20:11:54.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-10T20:11:54.583 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:11:54.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-10T20:11:54.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-10T20:11:54.759 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:11:54.775 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-10T20:11:54.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-10T20:11:54.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-10T20:11:55.206 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:11:55.206 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-09-10T20:11:37.461794+0000 47'102 51:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:08:56.472361+0000 0'0 2024-09-10T20:08:56.472361+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:06:37.671994+0000 0 0 2024-09-10T20:11:55.207 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:11:55.207 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:11:55.207 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:11:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-10T20:11:55.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-10T20:11:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:11:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-10T20:11:55.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:11:55.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-10T20:11:55.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:11:55.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-10T20:11:55.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:11:55.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-10T20:11:55.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-10T20:11:55.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:11:55.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-10T20:11:55.410 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-10T20:11:55.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:11:55.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-10T20:11:55.446 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-10T20:11:55.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-10T20:11:55.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-10T20:11:55.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:11:55.453 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4619.0:1 2024-09-10T20:09:30.788934+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:11:55.453 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4619.0:1 2024-09-10T20:09:30.788934+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4738.0:1 2024-09-10T20:10:33.312675+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4610.0:1 2024-09-10T20:09:30.768937+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4610.0:1 2024-09-10T20:09:30.768937+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4625.0:1 2024-09-10T20:09:30.794912+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:11:55.454 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4625.0:1 2024-09-10T20:09:30.794912+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4625.0:1 2024-09-10T20:09:30.794912+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4613.0:1 2024-09-10T20:09:30.775165+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4613.0:1 2024-09-10T20:09:30.775165+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:11:55.455 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4613.0:1 2024-09-10T20:09:30.775165+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-09-10T20:09:30.794174+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: keeping 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-09-10T20:09:30.794174+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:11:55.456 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4624.0:1 2024-09-10T20:09:30.794174+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-10T20:11:55.457 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4616.0:1 2024-09-10T20:09:30.781273+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4616.0:1 2024-09-10T20:09:30.781273+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:11:23.007+0000 7fc666d2e640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-10T20:11:55.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:11:55.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-10T20:11:55.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:11:55.465 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-10T20:11:55.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-10T20:11:55.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.27146.out 2024-09-10T20:11:55.467 INFO:tasks.workunit.client.0.smithi039.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27146.out': No such file or directory 2024-09-10T20:11:55.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-10T20:11:55.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-10T20:11:55.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-10T20:11:55.981 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-10T20:11:55.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-10T20:11:55.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:11:55.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:11:55.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:11:55.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:11:55.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:12:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:12:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-10T20:12:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:12:01.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:12:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:12:01.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:12:01.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:12:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:12:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:12:01.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:12:01.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:12:01.325 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:12:01.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:12:01.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:12:01.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:12:01.327 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:12:01.328 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:12:01.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:01.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:12:01.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:12:01.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:01.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:12:01.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:12:01.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:12:01.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:12:01.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:01.356 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:01.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:12:01.358 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:12:01.359 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:12:01.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:12:01.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:12:01.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:12:01.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:12:01.360 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:12:01.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:12:01.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:12:01.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:12:01.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:12:01.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:12:01.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:01.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:12:01.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:12:01.364 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:12:01.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:12:01.365 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:12:01.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:12:01.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:12:01.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:01.367 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:01.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:12:01.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:12:01.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:12:01.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-10T20:12:01.369 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:12:01.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:01.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:01.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27146 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-10T20:12:01.372 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-10T20:12:01.373 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-10T20:12:01.373 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-10T20:12:01.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-10T20:12:01.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-10T20:12:01.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:12:01.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:12:01.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:12:01.831 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:12:01.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:01.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:01.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:12:01.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:12:02.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:12:02.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:12:02.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:12:02.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:12:02.033 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:12:02.034 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:12:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:12:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:12:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:12:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:12:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:02.035 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:12:02.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:12:02.035 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get fsid 2024-09-10T20:12:02.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:12:02.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:12:02.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:12:02.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:12:02.158 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:12:02.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:12:02.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:12:02.158 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:12:02.158 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:12:02.159 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:02.159 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:02.159 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:12:02.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:12:02.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get mon_host 2024-09-10T20:12:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-10T20:12:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-10T20:12:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:12:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:12:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:12:02.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-10T20:12:02.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:12:03.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:12:03.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:12:03.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:12:03.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:12:03.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:03.184 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:03.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:12:03.184 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:12:03.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:12:03.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:12:03.249 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:12:03.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:12:03.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:12:03.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:12:03.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:12:03.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:12:03.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=11e1e427-b96e-40c4-8c63-b5bfbbef0271 2024-09-10T20:12:03.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 11e1e427-b96e-40c4-8c63-b5bfbbef0271' 2024-09-10T20:12:03.252 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 11e1e427-b96e-40c4-8c63-b5bfbbef0271 2024-09-10T20:12:03.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:12:03.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQATqOBmiCtjEBAAn5Q2clTMxTLVIBH9/qmFKQ== 2024-09-10T20:12:03.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQATqOBmiCtjEBAAn5Q2clTMxTLVIBH9/qmFKQ=="}' 2024-09-10T20:12:03.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 11e1e427-b96e-40c4-8c63-b5bfbbef0271 -i td/divergent-priors/0/new.json 2024-09-10T20:12:03.611 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:12:03.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-10T20:12:03.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATqOBmiCtjEBAAn5Q2clTMxTLVIBH9/qmFKQ== --osd-uuid 11e1e427-b96e-40c4-8c63-b5bfbbef0271 2024-09-10T20:12:03.680 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:03.679+0000 7f49c29007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:03.696 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:03.695+0000 7f49c29007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:03.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:03.699+0000 7f49c29007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:03.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:03.699+0000 7f49c29007c0 -1 bdev(0x561517395c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:12:03.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:03.699+0000 7f49c29007c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-10T20:12:09.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-10T20:12:09.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:12:09.529 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:12:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:12:09.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:12:10.147 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:12:10.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:12:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:12:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:12:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:12:10.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:12:10.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:10.179+0000 7f918e9827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:10.183 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:10.183+0000 7f918e9827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:10.184 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:10.183+0000 7f918e9827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:10.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:12:10.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:12:10.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:12:10.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:12:10.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:12:10.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:10.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:12:10.666 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:12:10.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:10.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:11.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:11.689 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:11.687+0000 7f918e9827c0 -1 Falling back to public interface 2024-09-10T20:12:12.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:12.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:12.158 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:12:12.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:12:12.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:12.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:12.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:13.441 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:13.439+0000 7f918e9827c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:12:13.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:13.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:13.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:12:13.662 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:12:13.662 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:13.663 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:14.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:15.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:15.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:15.353 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:12:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:12:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:15.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:15.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:16.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:16.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:16.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:12:16.821 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:12:16.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:16.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:17.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:18.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:18.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:18.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:12:18.363 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:12:18.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:18.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:18.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:19.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:19.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:19.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:12:19.855 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:12:19.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:19.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:20.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:21.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:21.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:21.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:12:21.374 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:12:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:21.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:21.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:22.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:22.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:22.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:12:22.857 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:12:22.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:22.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:23.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:24.357 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:12:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:12:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:24.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:25.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:25.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:25.873 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:12:25.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:12:25.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:25.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:26.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:12:27.378 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:12:27.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:27.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:28.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:29.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:12:29.389 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:12:29.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:29.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:29.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:30.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:30.967 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:30.967 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:12:30.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:12:30.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:30.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:32.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:33.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:33.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:33.086 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:12:33.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:12:33.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:33.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:33.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:34.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:34.997 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:34.997 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:12:34.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:12:34.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:34.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:12:35.480 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/558287755,v1:127.0.0.1:6803/558287755] [v2:127.0.0.1:6804/558287755,v1:127.0.0.1:6805/558287755] exists,up 11e1e427-b96e-40c4-8c63-b5bfbbef0271 2024-09-10T20:12:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:12:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:12:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:12:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-10T20:12:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-10T20:12:35.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:12:35.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:12:35.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:12:35.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:12:35.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:12:35.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:12:35.487 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:12:35.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f6d81b4e-47a0-4251-8ee2-fe937090de25' 2024-09-10T20:12:35.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:12:35.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzqOBmM3CBHhAA39deBnAKf4hEHBL4R+Z7IQ== 2024-09-10T20:12:35.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzqOBmM3CBHhAA39deBnAKf4hEHBL4R+Z7IQ=="}' 2024-09-10T20:12:35.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6d81b4e-47a0-4251-8ee2-fe937090de25 -i td/divergent-priors/1/new.json 2024-09-10T20:12:36.043 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:12:36.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-10T20:12:36.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzqOBmM3CBHhAA39deBnAKf4hEHBL4R+Z7IQ== --osd-uuid f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:12:36.105 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:36.103+0000 7f99566047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:36.124 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:36.123+0000 7f99566047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:36.125 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:36.123+0000 7f99566047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:36.125 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:36.123+0000 7f99566047c0 -1 bdev(0x564c452f5c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:12:36.125 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:36.127+0000 7f99566047c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-10T20:12:42.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-10T20:12:42.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:12:42.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:12:42.236 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:12:42.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:12:42.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:12:42.881 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:12:42.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:12:42.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:12:42.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:12:42.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:12:42.921 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:42.918+0000 7fafd6c507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:42.923 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:42.922+0000 7fafd6c507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:42.924 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:42.922+0000 7fafd6c507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:12:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:12:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:12:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:12:43.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:43.353 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:43.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:43.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:44.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:44.422+0000 7fafd6c507c0 -1 Falling back to public interface 2024-09-10T20:12:44.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:44.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:44.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:12:44.855 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:12:44.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:44.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:45.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:46.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:12:46.206+0000 7fafd6c507c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:12:46.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:46.352 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:12:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:12:46.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:46.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:46.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:47.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:47.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:47.910 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:12:47.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:12:47.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:47.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:48.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:49.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:49.366 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:12:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:12:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:49.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:49.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:50.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:50.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:50.834 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:12:50.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:12:50.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:50.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:51.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:52.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:52.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:52.376 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:12:52.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:12:52.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:52.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:53.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:54.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:54.022 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:12:54.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:12:54.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:54.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:54.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:55.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:55.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:55.487 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:12:55.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:12:55.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:55.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:56.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:57.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:57.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:57.034 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:12:57.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:12:57.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:57.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:57.501 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:12:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:12:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:12:58.503 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:12:58.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:12:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:12:58.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:12:59.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:00.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:00.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:00.651 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:13:00.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:13:00.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:00.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:13:01.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:02.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:02.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:02.855 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:13:02.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:13:02.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:02.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:13:03.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:04.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:04.861 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:13:04.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:13:04.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:04.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:13:05.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:06.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:06.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:06.390 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:13:06.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:13:06.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:06.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:13:07.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:08.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:08.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:08.575 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:13:08.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:13:08.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:08.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:13:09.073 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3965607747,v1:127.0.0.1:6811/3965607747] [v2:127.0.0.1:6812/3965607747,v1:127.0.0.1:6813/3965607747] exists,up f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:13:09.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:13:09.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:13:09.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:13:09.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:13:09.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:13:09.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:13:09.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0205c44a-8881-450b-ac91-9a38b8066f16 2024-09-10T20:13:09.080 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 0205c44a-8881-450b-ac91-9a38b8066f16 2024-09-10T20:13:09.081 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 0205c44a-8881-450b-ac91-9a38b8066f16' 2024-09-10T20:13:09.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:13:09.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBVqOBma42UBhAA3SRVChHkg/wBugk3fuyEKQ== 2024-09-10T20:13:09.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBVqOBma42UBhAA3SRVChHkg/wBugk3fuyEKQ=="}' 2024-09-10T20:13:09.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0205c44a-8881-450b-ac91-9a38b8066f16 -i td/divergent-priors/2/new.json 2024-09-10T20:13:09.634 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:13:09.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-10T20:13:09.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVqOBma42UBhAA3SRVChHkg/wBugk3fuyEKQ== --osd-uuid 0205c44a-8881-450b-ac91-9a38b8066f16 2024-09-10T20:13:09.698 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:09.698+0000 7f3022b3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:13:09.705 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:09.702+0000 7f3022b3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:13:09.706 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:09.706+0000 7f3022b3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:13:09.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:09.706+0000 7f3022b3f7c0 -1 bdev(0x5646d8e3fc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:13:09.707 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:09.706+0000 7f3022b3f7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-10T20:13:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-10T20:13:15.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:13:15.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:13:15.764 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:13:15.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:13:16.370 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:13:16.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:13:16.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:13:16.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:13:16.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:13:16.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:13:16.413 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:16.410+0000 7f1e6ebf67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:13:16.415 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:16.414+0000 7f1e6ebf67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:13:16.416 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:16.414+0000 7f1e6ebf67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:13:16.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:13:16.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:13:16.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:13:16.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:13:16.857 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:13:16.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:13:16.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:16.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:13:16.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:16.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:17.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:17.432 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:17.430+0000 7f1e6ebf67c0 -1 Falling back to public interface 2024-09-10T20:13:18.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:18.358 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:18.358 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:13:18.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:13:18.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:18.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:18.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:19.072 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:13:19.070+0000 7f1e6ebf67c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:13:19.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:19.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:19.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:13:19.811 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:13:19.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:19.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:20.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:21.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:21.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:21.526 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:13:21.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:13:21.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:21.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:22.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:23.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:23.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:23.042 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:13:23.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:13:23.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:23.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:23.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:24.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:24.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:24.505 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:13:24.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:13:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:24.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:25.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:26.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:26.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:26.164 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:13:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:13:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:26.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:26.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:27.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:27.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:27.791 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:13:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:13:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:27.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:28.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:29.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:29.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:29.258 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:13:29.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:13:29.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:29.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:29.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:30.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:30.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:30.727 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:13:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:13:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:30.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:31.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:32.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:32.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:32.190 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:13:32.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:13:32.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:32.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:33.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:34.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:34.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:34.031 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:13:34.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:13:34.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:34.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:34.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:35.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:35.819 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:35.819 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:13:35.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:13:35.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:35.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:36.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:37.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:37.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:37.641 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:13:37.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:13:37.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:37.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:38.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:39.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:39.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:39.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:13:39.419 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:13:39.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:39.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:40.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:41.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:41.234 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:13:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:13:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:41.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:41.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:13:42.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:13:42.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:13:42.926 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:13:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:13:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:13:42.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:13:43.389 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/16103251,v1:127.0.0.1:6819/16103251] [v2:127.0.0.1:6820/16103251,v1:127.0.0.1:6821/16103251] exists,up 0205c44a-8881-450b-ac91-9a38b8066f16 2024-09-10T20:13:43.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:13:43.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:13:43.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:13:43.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-10T20:13:44.085 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-10T20:13:44.104 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-10T20:13:44.898 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-10T20:13:44.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-10T20:13:45.788 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-10T20:13:45.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-10T20:13:45.812 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T20:13:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-10T20:13:46.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:13:47.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-10T20:13:48.399 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 size to 3 2024-09-10T20:13:48.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-10T20:13:49.170 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 min_size to 2 2024-09-10T20:13:49.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-10T20:13:49.873 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 pg_autoscale_mode to on 2024-09-10T20:13:49.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-10T20:13:49.896 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-10T20:13:49.897 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1725999229 2024-09-10T20:13:49.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-10T20:13:49.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-10T20:13:49.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-10T20:13:49.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:13:49.898 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:13:50.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:13:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:13:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:13:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:13:50.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:13:50.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:13:50.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-10T20:13:50.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-10T20:13:50.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-10T20:13:50.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:13:50.537 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:13:50.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672974 2024-09-10T20:13:50.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672974 2024-09-10T20:13:50.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672974' 2024-09-10T20:13:50.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:13:50.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:13:50.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-10T20:13:50.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-10T20:13:50.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672974 2-64424509448' 2024-09-10T20:13:50.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:13:50.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-10T20:13:50.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:13:50.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:13:50.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-10T20:13:50.881 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:13:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-10T20:13:50.882 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836502 2024-09-10T20:13:50.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-10T20:13:50.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:13:51.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-10T20:13:51.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:13:52.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:13:52.385 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:13:52.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836502 2024-09-10T20:13:52.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:13:52.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672974 2024-09-10T20:13:52.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:13:52.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:13:52.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:13:52.897 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672974 2024-09-10T20:13:52.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672974 2024-09-10T20:13:52.898 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672974 2024-09-10T20:13:52.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-10T20:13:52.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:13:53.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-10T20:13:53.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:13:53.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-10T20:13:53.338 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:13:53.339 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:13:53.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:13:53.340 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-10T20:13:53.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-10T20:13:53.341 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509448 2024-09-10T20:13:53.341 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-10T20:13:53.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:13:53.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-10T20:13:53.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-10T20:13:53.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:13:53.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:13:53.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:13:53.840 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:13:53.840 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:13:53.841 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:13:53.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:13:53.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:13:53.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:13:53.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:13:53.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:13:53.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:13:53.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:13:53.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:13:53.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:13:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:13:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:13:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:13:54.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:13:54.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:13:54.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:13:54.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-10T20:13:54.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-10T20:13:54.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-10T20:13:54.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:13:54.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:13:54.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672977 2024-09-10T20:13:54.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672977 2024-09-10T20:13:54.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-09-10T20:13:54.815 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:13:54.815 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:13:54.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-10T20:13:54.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-10T20:13:54.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509451' 2024-09-10T20:13:54.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:13:54.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-10T20:13:54.983 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:13:54.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:13:54.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-10T20:13:54.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:13:54.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-10T20:13:54.987 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836505 2024-09-10T20:13:54.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-10T20:13:54.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:13:55.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836505 2024-09-10T20:13:55.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:13:56.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:13:56.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:13:56.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-10T20:13:56.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:13:56.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672977 2024-09-10T20:13:56.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:13:56.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:13:56.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672977 2024-09-10T20:13:56.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:13:56.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672977 2024-09-10T20:13:56.982 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672977 2024-09-10T20:13:56.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-10T20:13:56.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:13:57.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-10T20:13:57.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:13:57.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-10T20:13:57.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:13:57.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:13:57.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-10T20:13:57.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:13:57.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-10T20:13:57.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-10T20:13:57.464 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509451 2024-09-10T20:13:57.464 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:13:57.954 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-10T20:13:57.954 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:13:57.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:13:57.955 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:13:58.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:13:58.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:13:58.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:13:58.618 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:13:58.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:13:58.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:13:58.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:13:58.619 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:13:59.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:13:59.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:13:59.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:13:59.053 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:13:59.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:13:59.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:13:59.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:13:59.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-10T20:13:59.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-10T20:14:00.054 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:14:00.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-10T20:14:00.070 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-10T20:14:00.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-10T20:14:00.071 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:14:00.475 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:14:00.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:13:47.489915+0000 0'0 31:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T00:55:53.897724+0000 0 0 2024-09-10T20:14:00.476 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:14:00.476 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:14:00.476 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:14:00.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-10T20:14:00.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1725999240 2024-09-10T20:14:00.494 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1725999240 - 1725999229 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=11 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 11 -gt 300 ']' 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-10T20:14:00.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-10T20:14:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-10T20:14:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-10T20:14:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-10T20:14:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-10T20:14:00.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-10T20:14:00.497 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-10T20:14:00.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-10T20:14:00.498 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-10T20:14:00.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:00.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-10T20:14:00.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:00.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-10T20:14:00.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:00.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-10T20:14:01.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:01.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-10T20:14:01.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:01.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-10T20:14:01.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:01.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-10T20:14:01.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:01.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-10T20:14:01.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:01.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-10T20:14:01.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:01.851 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-10T20:14:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:02.034 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-10T20:14:02.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:02.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-10T20:14:02.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:02.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-10T20:14:02.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:02.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-10T20:14:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:02.731 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-10T20:14:02.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:02.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-10T20:14:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:03.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-10T20:14:03.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:03.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-10T20:14:03.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:03.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-10T20:14:03.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:03.551 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-10T20:14:03.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:03.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-10T20:14:03.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:03.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-10T20:14:04.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:04.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-10T20:14:04.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:04.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-10T20:14:04.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:04.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-10T20:14:04.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:04.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-10T20:14:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:04.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-10T20:14:04.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:04.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-10T20:14:05.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:05.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-10T20:14:05.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:05.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-10T20:14:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:05.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-10T20:14:05.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:05.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-10T20:14:05.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:05.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-10T20:14:06.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:06.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-10T20:14:06.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:06.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-10T20:14:06.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:06.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-10T20:14:06.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:06.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-10T20:14:06.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:06.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-10T20:14:06.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-10T20:14:07.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:07.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-10T20:14:07.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:07.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-10T20:14:07.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:07.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-10T20:14:07.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:07.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-10T20:14:07.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:07.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-10T20:14:07.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:07.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-10T20:14:08.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:08.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-10T20:14:08.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:08.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-10T20:14:08.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:08.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-10T20:14:08.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:08.541 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-10T20:14:08.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:08.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-10T20:14:09.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:09.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-10T20:14:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-10T20:14:09.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:09.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-10T20:14:09.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:09.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-10T20:14:09.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:09.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-10T20:14:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:09.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-10T20:14:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:10.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-10T20:14:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:10.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-10T20:14:10.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:10.453 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-10T20:14:11.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:11.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-10T20:14:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:11.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-10T20:14:11.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:11.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-10T20:14:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:12.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-10T20:14:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:12.346 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-10T20:14:12.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:12.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-10T20:14:12.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:12.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-10T20:14:12.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:12.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-10T20:14:13.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:13.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-10T20:14:13.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:13.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-10T20:14:13.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:13.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-10T20:14:13.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:13.672 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-10T20:14:13.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:13.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-10T20:14:14.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:14.032 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-10T20:14:14.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:14.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-10T20:14:14.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:14.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-10T20:14:14.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:14.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-10T20:14:15.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:15.135 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-10T20:14:15.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:15.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-10T20:14:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:15.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-10T20:14:16.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:16.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-10T20:14:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:16.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-10T20:14:16.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:16.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-10T20:14:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:16.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-10T20:14:17.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:17.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-10T20:14:18.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:18.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-10T20:14:18.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:18.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-10T20:14:18.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:18.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-10T20:14:19.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:19.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-10T20:14:19.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:19.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-10T20:14:19.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:19.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-10T20:14:20.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:20.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-10T20:14:20.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:20.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-10T20:14:21.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:21.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-10T20:14:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:21.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-10T20:14:21.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:21.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-10T20:14:21.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:21.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-10T20:14:22.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:22.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-10T20:14:22.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:22.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-10T20:14:22.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:22.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-10T20:14:22.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:22.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-10T20:14:23.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-10T20:14:23.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-10T20:14:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:14:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-10T20:14:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:14:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:14:23.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:14:23.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:14:23.531 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:14:23.531 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:14:23.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:14:23.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:14:23.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:14:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:14:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:14:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:14:23.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:14:23.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:14:23.615 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:14:24.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:14:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:14:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:14:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:14:24.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:14:24.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:14:24.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836513 2024-09-10T20:14:24.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836513 2024-09-10T20:14:24.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513' 2024-09-10T20:14:24.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:14:24.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:14:24.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672985 2024-09-10T20:14:24.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672985 2024-09-10T20:14:24.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672985' 2024-09-10T20:14:24.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:14:24.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:14:24.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-10T20:14:24.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-10T20:14:24.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836513 1-42949672985 2-64424509459' 2024-09-10T20:14:24.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:14:24.946 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836513 2024-09-10T20:14:24.947 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:14:24.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:14:24.949 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836513 2024-09-10T20:14:24.950 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:14:24.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836513 2024-09-10T20:14:24.951 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836513 2024-09-10T20:14:24.952 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-10T20:14:24.952 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:14:25.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836511 -lt 21474836513 2024-09-10T20:14:25.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:14:26.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:14:26.468 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:14:26.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836514 -lt 21474836513 2024-09-10T20:14:26.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:14:26.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:14:26.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672985 2024-09-10T20:14:26.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:14:26.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672985 2024-09-10T20:14:26.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:14:26.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672985 2024-09-10T20:14:26.941 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-10T20:14:26.942 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672985 2024-09-10T20:14:26.942 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:14:27.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-10T20:14:27.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:14:27.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-10T20:14:27.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:14:27.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:14:27.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-10T20:14:27.387 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:14:27.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-10T20:14:27.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-10T20:14:27.388 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509459 2024-09-10T20:14:27.388 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:14:27.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-10T20:14:27.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:14:27.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:14:27.836 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:14:28.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-10T20:14:28.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:14:28.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:14:28.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:14:28.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:14:28.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:14:28.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:14:28.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:14:28.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=31 2024-09-10T20:14:28.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:14:28.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:14:28.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:14:29.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 31 = 32 2024-09-10T20:14:29.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 31 '!=' -1 2024-09-10T20:14:29.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:14:29.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=31 2024-09-10T20:14:29.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:14:29.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:14:29.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:14:29.531 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:14:29.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:14:29.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:14:29.531 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:14:29.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:14:29.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:14:29.532 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:14:29.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=31 2024-09-10T20:14:29.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:14:29.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:14:29.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:14:30.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 31 = 32 2024-09-10T20:14:30.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 31 '!=' 31 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:14:30.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:14:31.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:14:31.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:14:31.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:14:31.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:14:31.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:14:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:14:31.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:14:31.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:14:31.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:14:31.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:14:31.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:14:31.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:14:31.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:14:31.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=32 2024-09-10T20:14:31.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:14:31.963 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:14:31.964 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:14:32.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 = 32 2024-09-10T20:14:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:14:32.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:14:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-10T20:14:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-10T20:14:32.703 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-10T20:14:32.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-10T20:14:32.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:14:33.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-10T20:14:33.183 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-10T20:14:33.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-10T20:14:33.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:14:33.663 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:14:33.663 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:21.539536+0000 34'58 44:149 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:23:32.622738+0000 3 0 2024-09-10T20:14:33.663 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:25.429683+0000 42'59 44:146 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:42:14.539510+0000 2 0 2024-09-10T20:14:33.663 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:14:18.312463+0000 34'59 44:140 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:32:03.780403+0000 4 0 2024-09-10T20:14:33.663 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:22.467539+0000 41'58 44:133 [1,2,0] 1 [1,2,0] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:35:22.891469+0000 3 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:14:19.124149+0000 32'57 45:133 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:05:01.119090+0000 3 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:14:19.358467+0000 40'59 44:137 [2,0,1] 2 [2,0,1] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:09:43.792818+0000 4 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 active+clean 2024-09-10T20:14:19.540782+0000 41'60 44:135 [1,0,2] 1 [1,0,2] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:49:55.128654+0000 2 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:14:23.498344+0000 32'57 44:132 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:12:43.949738+0000 2 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2468 0 0 18 42 18 active+clean 2024-09-10T20:14:25.648254+0000 36'60 44:148 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:18:56.734827+0000 4 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 active+clean 2024-09-10T20:14:20.339370+0000 40'60 44:141 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:00:48.424893+0000 5 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:14:16.430610+0000 34'58 44:133 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:16:03.823635+0000 2 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:14:24.566118+0000 42'59 44:135 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:57:59.304104+0000 5 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 active+clean 2024-09-10T20:14:21.045685+0000 35'58 45:140 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:06:35.080533+0000 1 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:17.393030+0000 39'58 44:133 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:41:25.175663+0000 2 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 active+clean 2024-09-10T20:14:15.602622+0000 38'59 44:140 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:31:00.397747+0000 0 0 2024-09-10T20:14:33.664 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:14:18.414385+0000 32'57 44:132 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:51:58.948036+0000 2 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:12.531901+0000 41'58 44:133 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:57:04.269026+0000 2 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:21.296777+0000 40'59 44:137 [2,0,1] 2 [2,0,1] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:15:19.918624+0000 3 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:14:27.538085+0000 39'59 44:143 [1,0,2] 1 [1,0,2] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:01:23.266453+0000 4 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:14:28.517817+0000 34'58 44:135 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:43:19.464303+0000 4 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1716 0 0 16 42 16 active+clean 2024-09-10T20:14:11.473980+0000 41'58 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 queued for scrub 0 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:14:22.217210+0000 32'57 44:135 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:53:39.544700+0000 3 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:14:23.028344+0000 32'57 45:133 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:38:27.700044+0000 1 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 4 0 0 0 0 2333 0 0 17 42 17 active+clean 2024-09-10T20:14:19.927879+0000 41'59 44:129 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 queued for scrub 0 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 4 0 0 0 0 2333 0 0 17 42 17 active+clean 2024-09-10T20:14:11.470068+0000 41'59 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 queued for scrub 0 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:23.241603+0000 34'58 44:138 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:47:04.000562+0000 3 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:14:26.540254+0000 34'59 44:137 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:48:31.015178+0000 4 0 2024-09-10T20:14:33.665 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 2 0 0 0 0 1099 0 0 15 42 15 active+clean 2024-09-10T20:14:19.810341+0000 32'57 44:124 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 queued for scrub 0 0 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 0 0 0 2333 0 0 17 42 17 active+clean 2024-09-10T20:14:11.476760+0000 41'59 44:126 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 queued for scrub 0 0 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:14:15.601526+0000 32'57 44:138 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:06:00.595941+0000 2 0 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:14:24.060257+0000 38'58 45:134 [0,1,2] 0 [0,1,2] 0 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:06:48.933426+0000 4 0 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:14:20.451881+0000 41'59 44:147 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:49:20.067051+0000 4 0 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:14:33.666 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:14:33.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:33.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-10T20:14:33.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:14:33.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:14:33.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:14:33.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:14:33.680 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:14:33.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:14:33.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-10T20:14:33.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-10T20:14:33.779 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:14:33.779 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:14:33.779 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:14:33.790 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:33.791 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-10T20:14:33.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:14:33.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:14:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:14:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:14:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:14:33.793 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:14:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-10T20:14:33.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-10T20:14:33.902 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:14:33.902 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:14:33.902 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:14:33.914 INFO:tasks.workunit.client.0.smithi039.stdout:create new divergent objects 2024-09-10T20:14:33.914 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-10T20:14:33.915 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-10T20:14:33.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:14:33.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-10T20:14:33.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:14:33.918 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent objects 2024-09-10T20:14:33.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-10T20:14:33.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-10T20:14:33.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-10T20:14:33.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-10T20:14:33.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-10T20:14:33.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-10T20:14:33.921 INFO:tasks.workunit.client.0.smithi039.stdout:remove divergent objects 2024-09-10T20:14:33.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-10T20:14:33.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-10T20:14:33.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-10T20:14:33.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-10T20:14:33.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-10T20:14:33.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-10T20:14:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-10T20:14:33.926 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-10T20:14:33.926 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-10T20:14:33.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-10T20:14:33.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-10T20:14:33.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-10T20:14:43.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-10T20:14:43.935 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-10T20:14:43.935 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-10T20:14:43.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-10T20:14:43.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:21.539536+0000 34'58 53:172 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:53:44.662371+0000 3 0 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:25.429683+0000 42'59 53:169 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:53:41.056847+0000 2 0 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:14:18.312463+0000 34'59 53:163 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:28:41.271218+0000 4 0 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:39.012490+0000 41'58 53:162 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:38:23.581160+0000 3 0 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:14:37.778307+0000 32'57 53:140 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:37.778164+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:03:06.844089+0000 3 0 2024-09-10T20:14:44.374 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 0 0 0 3435 0 0 18 42 18 active+clean 2024-09-10T20:14:35.711130+0000 47'60 53:174 [1,2,0] 1 [1,2,0] 1 47'60 2024-09-10T20:14:35.710973+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:27:58.448747+0000 5 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 active+clean 2024-09-10T20:14:33.386090+0000 41'60 53:164 [1,2,0] 1 [1,2,0] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:19:38.519067+0000 2 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:14:23.498344+0000 32'57 53:155 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:32:53.205958+0000 2 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2468 0 0 18 42 18 active+clean 2024-09-10T20:14:25.648254+0000 36'60 53:171 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:53:31.093098+0000 4 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 active+clean 2024-09-10T20:14:20.339370+0000 40'60 53:164 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:49:52.083179+0000 5 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:14:16.430610+0000 34'58 53:156 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:49:42.022180+0000 2 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:14:24.566118+0000 42'59 53:158 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:42:49.498726+0000 5 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 active+clean 2024-09-10T20:14:21.045685+0000 35'58 53:160 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:13:42.008835+0000 1 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:17.393030+0000 39'58 53:156 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:12:38.509699+0000 2 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 active+clean 2024-09-10T20:14:15.602622+0000 38'59 53:163 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:59:17.714562+0000 0 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:14:18.414385+0000 32'57 53:155 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:47:46.096874+0000 2 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:12.531901+0000 41'58 53:156 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:59:17.389755+0000 2 0 2024-09-10T20:14:44.375 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:21.296777+0000 40'59 53:160 [2,0,1] 2 [2,0,1] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:29:34.001860+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 0 0 0 1851 0 0 19 42 19 active+clean 2024-09-10T20:14:27.538085+0000 47'61 53:168 [1,0,2] 1 [1,0,2] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:29:35.023276+0000 4 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:14:28.517817+0000 34'58 53:158 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:15:43.107641+0000 4 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:29.559788+0000 41'58 53:156 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:05:49.756014+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:14:22.217210+0000 47'58 53:159 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:07:52.872272+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:14:23.028344+0000 32'57 53:153 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:17:32.415252+0000 1 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:30.573932+0000 41'59 53:160 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:23:17.652063+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:31.602308+0000 41'59 53:157 [1,0,2] 1 [1,0,2] 1 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:11:37.125311+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:14:23.241603+0000 34'58 53:161 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:36:21.194974+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:14:26.540254+0000 34'59 53:160 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:54:18.389025+0000 4 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:14:33.683819+0000 32'57 53:155 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:23.303956+0000 1 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:14:34.706027+0000 41'59 53:166 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:14:34.705861+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:05:57.621780+0000 3 0 2024-09-10T20:14:44.376 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:14:34.842940+0000 32'57 53:134 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:20:48.830761+0000 2 0 2024-09-10T20:14:44.377 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:14:38.742028+0000 38'58 53:150 [1,2,0] 1 [1,2,0] 1 38'58 2024-09-10T20:14:38.741886+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:56:39.712461+0000 4 0 2024-09-10T20:14:44.377 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 6 0 0 0 0 3435 0 0 19 42 19 active+clean 2024-09-10T20:14:38.924401+0000 47'61 53:178 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:56:21.304118+0000 4 0 2024-09-10T20:14:44.377 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:14:44.377 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:14:44.377 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:14:44.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-10T20:14:44.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:14:44.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:14:44.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:14:44.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:14:44.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:14:44.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:14:44.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-10T20:14:44.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-10T20:14:45.069 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:14:45.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-10T20:14:45.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-10T20:14:45.654 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:14:45.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-10T20:14:45.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-10T20:14:46.212 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:14:46.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:46.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-10T20:14:46.911 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-10T20:14:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:46.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-10T20:14:47.516 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-10T20:14:47.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-10T20:14:47.534 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-10T20:14:47.534 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:14:47.967 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:14:47.967 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:21.539536+0000 34'58 53:172 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:53:44.662371+0000 3 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:25.429683+0000 42'59 53:169 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:53:41.056847+0000 2 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:18.312463+0000 34'59 53:163 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:28:41.271218+0000 4 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:39.012490+0000 41'58 53:162 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:38:23.581160+0000 3 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 stale+active+clean 2024-09-10T20:14:37.778307+0000 32'57 53:140 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:37.778164+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:03:06.844089+0000 3 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 0 0 0 3435 0 0 18 42 18 stale+active+clean 2024-09-10T20:14:35.711130+0000 47'60 53:174 [1,2,0] 1 [1,2,0] 1 47'60 2024-09-10T20:14:35.710973+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:27:58.448747+0000 5 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 stale+active+clean 2024-09-10T20:14:33.386090+0000 41'60 53:164 [1,2,0] 1 [1,2,0] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:19:38.519067+0000 2 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 stale+active+clean 2024-09-10T20:14:23.498344+0000 32'57 53:155 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:32:53.205958+0000 2 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2468 0 0 18 42 18 stale+active+clean 2024-09-10T20:14:25.648254+0000 36'60 53:171 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:53:31.093098+0000 4 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 stale+active+clean 2024-09-10T20:14:20.339370+0000 40'60 53:164 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:49:52.083179+0000 5 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:16.430610+0000 34'58 53:156 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:49:42.022180+0000 2 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:24.566118+0000 42'59 53:158 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:42:49.498726+0000 5 0 2024-09-10T20:14:47.968 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:21.045685+0000 35'58 53:160 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:13:42.008835+0000 1 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:17.393030+0000 39'58 53:156 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:12:38.509699+0000 2 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:15.602622+0000 38'59 53:163 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:59:17.714562+0000 0 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 stale+active+clean 2024-09-10T20:14:18.414385+0000 32'57 53:155 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:47:46.096874+0000 2 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:12.531901+0000 41'58 53:156 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:59:17.389755+0000 2 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:21.296777+0000 40'59 53:160 [2,0,1] 2 [2,0,1] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:29:34.001860+0000 3 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 0 0 0 1851 0 0 19 42 19 stale+active+clean 2024-09-10T20:14:27.538085+0000 47'61 53:168 [1,0,2] 1 [1,0,2] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:29:35.023276+0000 4 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:28.517817+0000 34'58 53:158 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:15:43.107641+0000 4 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:29.559788+0000 41'58 53:156 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:05:49.756014+0000 3 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 2 0 0 0 0 1234 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:22.217210+0000 47'58 53:159 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:07:52.872272+0000 3 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 stale+active+clean 2024-09-10T20:14:23.028344+0000 32'57 53:153 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:17:32.415252+0000 1 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 0 0 0 1851 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:30.573932+0000 41'59 53:160 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:23:17.652063+0000 3 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1851 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:31.602308+0000 41'59 53:157 [1,0,2] 1 [1,0,2] 1 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:11:37.125311+0000 3 0 2024-09-10T20:14:47.969 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:23.241603+0000 34'58 53:161 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:36:21.194974+0000 3 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:26.540254+0000 34'59 53:160 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:54:18.389025+0000 4 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 617 0 0 15 42 15 stale+active+clean 2024-09-10T20:14:33.683819+0000 32'57 53:155 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:20:23.303956+0000 1 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 3 0 0 0 0 1851 0 0 17 42 17 stale+active+clean 2024-09-10T20:14:34.706027+0000 41'59 53:166 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:14:34.705861+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:05:57.621780+0000 3 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 stale+active+clean 2024-09-10T20:14:34.842940+0000 32'57 53:134 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:20:48.830761+0000 2 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 stale+active+clean 2024-09-10T20:14:38.742028+0000 38'58 53:150 [1,2,0] 1 [1,2,0] 1 38'58 2024-09-10T20:14:38.741886+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:56:39.712461+0000 4 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 6 0 0 0 0 3435 0 0 19 42 19 stale+active+clean 2024-09-10T20:14:38.924401+0000 47'61 53:178 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:56:21.304118+0000 4 0 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:14:47.970 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:14:47.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:14:47.987 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:14:47.988 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:14:47.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:14:47.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:14:47.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:14:47.989 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:14:47.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:14:47.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-10T20:14:47.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:14:47.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:14:47.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:14:47.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:14:48.030 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:48.029+0000 7f0ccff197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:14:48.032 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:48.029+0000 7f0ccff197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:14:48.033 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:48.033+0000 7f0ccff197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:14:48.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:14:48.498 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:14:48.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:48.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:14:48.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:14:49.540 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:49.537+0000 7f0ccff197c0 -1 Falling back to public interface 2024-09-10T20:14:49.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:14:49.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:49.983 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:14:49.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:14:49.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:49.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:14:50.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:14:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:14:51.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:51.491 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:14:51.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:14:51.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:51.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:14:51.832 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:51.828+0000 7f0ccff197c0 -1 osd.0 47 log_to_monitors true 2024-09-10T20:14:51.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:14:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:14:52.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:52.959 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:14:52.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:14:52.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:52.959 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:14:53.470 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 64 up_thru 37 down_at 55 last_clean_interval [5,47) [v2:127.0.0.1:6802/1056624563,v1:127.0.0.1:6803/1056624563] [v2:127.0.0.1:6804/1056624563,v1:127.0.0.1:6805/1056624563] exists,up 11e1e427-b96e-40c4-8c63-b5bfbbef0271 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:14:53.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:14:53.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:14:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:14:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:14:53.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:14:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:14:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:14:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:14:53.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:14:53.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:14:53.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:14:53.477 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:14:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:14:53.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:14:53.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-10T20:14:53.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:14:53.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:14:53.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:14:53.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:14:53.512 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:53.508+0000 7f7eb92f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:14:53.528 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:53.524+0000 7f7eb92f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:14:53.529 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:53.528+0000 7f7eb92f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:14:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:14:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:14:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:14:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:14:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:14:54.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:54.014 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:14:54.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:14:54.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:14:54.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:54.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:14:54.776 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:54.772+0000 7f7eb92f17c0 -1 Falling back to public interface 2024-09-10T20:14:55.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:14:55.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:55.518 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:14:55.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:14:55.518 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:55.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:14:55.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:14:56.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:14:56.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:56.980 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:14:56.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:14:56.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:56.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:14:57.208 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:14:57.204+0000 7f7eb92f17c0 -1 osd.2 47 log_to_monitors true 2024-09-10T20:14:57.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:14:58.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:14:58.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:14:58.461 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:14:58.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:14:58.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:14:58.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:14:58.984 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 69 up_thru 35 down_at 57 last_clean_interval [15,47) [v2:127.0.0.1:6810/3778029559,v1:127.0.0.1:6811/3778029559] [v2:127.0.0.1:6812/3778029559,v1:127.0.0.1:6813/3778029559] exists,up 0205c44a-8881-450b-ac91-9a38b8066f16 2024-09-10T20:14:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:14:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:14:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:14:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:58.985 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-10T20:14:59.557 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-10T20:14:59.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:14:59.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-10T20:15:00.763 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-10T20:15:00.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:15:00.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-10T20:15:00.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:15:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:15:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:15:00.788 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:15:00.788 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:15:00.789 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:15:00.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:15:00.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:15:00.789 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:15:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:15:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:15:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:15:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:15:00.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:15:00.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:15:01.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:15:01.575 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:15:01.575 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:15:01.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:15:01.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:15:01.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:15:01.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906948 2024-09-10T20:15:01.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906948 2024-09-10T20:15:01.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906948' 2024-09-10T20:15:01.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:15:01.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:15:02.068 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:15:02.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:15:02.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:15:02.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:15:02.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:15:02.069 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:15:02.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743427 2024-09-10T20:15:02.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743427 2024-09-10T20:15:02.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906948 2-296352743427' 2024-09-10T20:15:02.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:15:02.255 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-274877906948 2024-09-10T20:15:02.256 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:15:02.257 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:15:02.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-274877906948 2024-09-10T20:15:02.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:15:02.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906948 2024-09-10T20:15:02.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 274877906948' 2024-09-10T20:15:02.259 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 274877906948 2024-09-10T20:15:02.260 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:15:02.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906949 -lt 274877906948 2024-09-10T20:15:02.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:15:02.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-296352743427 2024-09-10T20:15:02.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:15:02.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:15:02.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-296352743427 2024-09-10T20:15:02.790 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:15:02.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743427 2024-09-10T20:15:02.792 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 296352743427 2024-09-10T20:15:02.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 296352743427' 2024-09-10T20:15:02.792 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:15:03.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743427 -lt 296352743427 2024-09-10T20:15:03.278 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:15:03.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:03.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:03.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-10T20:15:03.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:03.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:03.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:03.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:03.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:03.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:03.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:04.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:04.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:04.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:04.317 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:04.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:04.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:15:04.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:15:04.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:15:04.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:04.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:15:05.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:05.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:05.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:05.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:05.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:05.021 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:05.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:05.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:05.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:05.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:05.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:05.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:06.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:06.053 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:06.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:06.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:06.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:06.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:06.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:06.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:15:06.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:06.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:15:06.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:06.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:06.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:06.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:06.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:06.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:06.822 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:06.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:07.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:07.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:07.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:07.269 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:07.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:07.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:07.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:07.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:07.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:07.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:07.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:07.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:07.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:08.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:08.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:08.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:15:08.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:08.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:15:08.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:08.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:08.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:08.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:08.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:08.842 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:08.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:08.843 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:09.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:09.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:09.292 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:09.293 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:09.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:09.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:09.886 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:15:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:10.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:15:11.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:11.278 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:11.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:11.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:11.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:11.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:11.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:11.280 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:11.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:11.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:11.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:11.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:12.347 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:12.348 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:12.349 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:12.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:12.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:12.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:15:12.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:12.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:15:14.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:14.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:14.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:14.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:14.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:14.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:14.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:14.556 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:15.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:15.012 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:15.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:15.013 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:15.623 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:15.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:15.624 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:16.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:16.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:16.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:15:16.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:16.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:15:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:19.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:19.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:19.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:19.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:19.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:19.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:19.411 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:19.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:19.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:19.872 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:19.873 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:20.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:20.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:20.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:15:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:21.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:15:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:27.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:27.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:27.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:27.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:27.443 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:27.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:27.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:27.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:27.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:27.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:27.910 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:28.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:29.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:29.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:29.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:15:29.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:29.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:15:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:36.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:36.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:36.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:36.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:36.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:36.394 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:36.395 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:36.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:36.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:36.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:37.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:37.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:37.427 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:15:38.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "8a3964a9-1551-4cea-95e3-4290ddd16e3b", 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:15:38.578876+0000", 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:15:38.579 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:15:38.580 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:15:38.581 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 32 pgs degraded", 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 32 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.582 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.585 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.587 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.588 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:15:38.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:15:38.591 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:12:01.404918Z", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:12:01.404918Z", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:15:38.592 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:15:38.593 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.594 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:15:38.595 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 78, 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:12:02.326494+0000", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:15:01.909107+0000", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:14:57.577984+0000", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:15:00.108037+0000", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:15:38.596 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:15:38.597 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:13:46.168014+0000", 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:15:38.598 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "on", 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 32, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 21, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 32, 2024-09-10T20:15:38.599 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 32, 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 32, 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "74", 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:15:38.600 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "33", 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:15:38.601 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:15:38.602 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "score_type": "Fair distribution", 2024-09-10T20:15:38.603 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 1.7799999713897705, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 1.7799999713897705, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 1.190000057220459, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 1.190000057220459, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:15:38.604 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "11e1e427-b96e-40c4-8c63-b5bfbbef0271", 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 47, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 64, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 74, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 55, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.605 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.606 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.607 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.608 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1056624563", 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1056624563", 2024-09-10T20:15:38.609 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1056624563", 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1056624563", 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "f6d81b4e-47a0-4251-8ee2-fe937090de25", 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:15:38.610 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 52, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 56, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.611 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.612 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.613 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3965607747", 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3965607747", 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3965607747", 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3965607747", 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:15:38.615 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "0205c44a-8881-450b-ac91-9a38b8066f16", 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 47, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 69, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 76, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 57, 2024-09-10T20:15:38.616 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.617 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.618 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.619 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.620 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3778029559", 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3778029559", 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3778029559", 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3778029559", 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.621 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:14:44.889964+0000", 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:12:34.901905+0000", 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:14:45.486363+0000", 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:15:38.622 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:13:07.680417+0000", 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:14:46.048103+0000", 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:13:42.255582+0000", 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:15:38.623 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:15:38.624 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:15:38.625 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1056624563,v1:127.0.0.1:6805/1056624563]", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:15:38.626 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:15:38.627 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:12:08.744619Z", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1056624563,v1:127.0.0.1:6803/1056624563]", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:15:38.628 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1056624563,v1:127.0.0.1:6809/1056624563]", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1056624563,v1:127.0.0.1:6807/1056624563]", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:15:38.629 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3965607747,v1:127.0.0.1:6813/3965607747]", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:15:38.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:15:38.631 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:12:41.572217Z", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3965607747,v1:127.0.0.1:6811/3965607747]", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3965607747,v1:127.0.0.1:6817/3965607747]", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3965607747,v1:127.0.0.1:6815/3965607747]", 2024-09-10T20:15:38.632 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.633 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3778029559,v1:127.0.0.1:6813/3778029559]", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:15:38.634 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:13:15.126433Z", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:15:38.635 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3778029559,v1:127.0.0.1:6811/3778029559]", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3778029559,v1:127.0.0.1:6817/3778029559]", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3778029559,v1:127.0.0.1:6815/3778029559]", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:15:38.636 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:15:38.637 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [ 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "floor": 36 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 65 2024-09-10T20:15:38.638 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 69 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 78, 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:15:38.639 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:15:38.640 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.641 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:15:38.642 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.643 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:15:38.644 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.645 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.646 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:15:38.647 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.648 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.649 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:15:38.650 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.651 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.651 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.651 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.651 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:15:38.651 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:15:38.652 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:15:38.653 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-10T20:15:38.654 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:15:38.655 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:15:38.655 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:15:38.655 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:15:38.655 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:15:38.655 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:15:38.655 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.656 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:15:38.656 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.657 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:15:38.657 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:15:38.657 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:12:02:302432+0000", 2024-09-10T20:15:38.657 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:15:38.658 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:15:38.659 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.660 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:15:38.661 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:15:38.661 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.661 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-10T20:15:38.662 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 32, 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62182, 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:15:38.663 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:15:38.664 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:15:38.665 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:15:38.665 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:15:38.665 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 13, 2024-09-10T20:15:38.665 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 13, 2024-09-10T20:15:38.665 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:15:38.666 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:15:38.667 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:15:38.668 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:15:38.669 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 523, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 523, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:15:38.670 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84464, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2888, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488336, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:15:38.671 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036056064, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2957312, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 751660, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:15:38.672 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 97, 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 97, 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 97000000, 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 97000000 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:15:38.673 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84464, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2888, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:15:38.674 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488336, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036056064, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2957312, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 751660, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:15:38.675 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 97, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 97, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 97000000, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 97000000 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.676 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62182, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:15:38.677 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 13, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 13, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:15:38.678 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:15:38.679 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:15:38.680 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 184566, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 523, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 523, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.681 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 274877906956 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672989 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 296352743435 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.682 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 32 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 19, 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:15:38.683 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 13, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.684 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:14:44.253551+0000", 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 97, 2024-09-10T20:15:38.685 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 432, 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 591, 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:15:38.686 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:15:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:report 3564221220 2024-09-10T20:15:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:15:38.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-10T20:15:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-10T20:15:38.687 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-10T20:15:38.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-10T20:15:38.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:15:39.035 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:15:39.035 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.367576+0000 34'58 78:157 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:52:02.710344+0000 3 0 2024-09-10T20:15:39.035 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368058+0000 42'59 78:159 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:06:02.950434+0000 2 0 2024-09-10T20:15:39.035 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.369421+0000 34'59 78:138 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:50:21.167484+0000 4 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.266233+0000 41'58 78:143 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:51:38.070635+0000 3 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:02.061489+0000 32'57 78:128 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:55.584648+0000 3 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.061357+0000 40'59 78:138 [2,0] 2 [2,0] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:17:05.827200+0000 4 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 3 0 0 1851 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.488026+0000 41'60 78:157 [2,0] 2 [2,0] 2 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:24:22.645126+0000 2 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.977085+0000 32'57 78:139 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:23:09.257405+0000 2 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2468 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.364911+0000 36'60 78:166 [0,2] 0 [0,2] 0 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:30:24.359258+0000 4 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.487584+0000 40'60 78:141 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:25:11.278817+0000 5 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362640+0000 34'58 78:130 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:53:02.440421+0000 2 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.370354+0000 42'59 78:149 [2,0] 2 [2,0] 2 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:03:02.756244+0000 5 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.364829+0000 35'58 78:146 [0,2] 0 [0,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:57:36.818603+0000 1 0 2024-09-10T20:15:39.036 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.365134+0000 39'58 78:137 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:11:45.912858+0000 2 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488077+0000 38'59 78:134 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:19:02.438290+0000 0 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362517+0000 32'57 78:134 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:08:51.120229+0000 2 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.363402+0000 41'58 78:142 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:40:34.279604+0000 2 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368616+0000 40'59 78:144 [2,0] 2 [2,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:58:38.628580+0000 3 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368558+0000 39'59 78:159 [2,0] 2 [2,0] 2 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:31:13.413252+0000 4 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.369057+0000 34'58 78:151 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:52:04.039387+0000 4 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362770+0000 41'58 78:151 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:10:08.517960+0000 3 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.365481+0000 32'57 78:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:07:31.456366+0000 3 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362099+0000 32'57 78:142 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:21:18.619051+0000 1 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365931+0000 41'59 78:155 [0,2] 0 [0,2] 0 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:41:22.588379+0000 3 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488142+0000 41'59 78:147 [2,0] 2 [2,0] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:15:07.508816+0000 3 0 2024-09-10T20:15:39.037 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.413696+0000 34'58 78:143 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:45:39.663228+0000 3 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.362516+0000 34'59 78:155 [0,2] 0 [0,2] 0 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:58:06.455346+0000 4 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.369618+0000 32'57 78:156 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:33:43.963351+0000 1 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2333 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368029+0000 41'59 78:156 [2,0] 2 [2,0] 2 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 user requested scrub scheduled @ 2024-09-10T20:16:02.201477+0000 0 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.367551+0000 32'57 78:132 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:56:48.695621+0000 2 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.061528+0000 38'58 78:138 [2,0] 2 [2,0] 2 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:37:36.538557+0000 4 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365641+0000 41'59 78:156 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:20.398497+0000 4 0 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:15:39.038 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:15:39.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-10T20:15:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:15:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-10T20:15:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:15:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:15:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:15:39.164 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:15:39.164 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:15:39.164 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:15:39.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:15:39.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:15:39.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:15:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:15:39.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:15:39.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:15:39.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:15:39.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:15:39.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:15:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:15:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:15:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:15:39.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:15:39.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:15:39.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:15:39.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=274877906958 2024-09-10T20:15:39.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 274877906958 2024-09-10T20:15:39.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906958' 2024-09-10T20:15:39.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:15:39.919 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:15:40.054 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:15:40.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:15:40.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:15:40.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:15:40.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:15:40.057 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:15:40.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=296352743437 2024-09-10T20:15:40.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 296352743437 2024-09-10T20:15:40.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-274877906958 2-296352743437' 2024-09-10T20:15:40.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:15:40.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-274877906958 2024-09-10T20:15:40.229 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:15:40.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:15:40.230 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-274877906958 2024-09-10T20:15:40.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:15:40.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=274877906958 2024-09-10T20:15:40.232 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 274877906958 2024-09-10T20:15:40.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 274877906958' 2024-09-10T20:15:40.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:15:40.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 274877906958 -lt 274877906958 2024-09-10T20:15:40.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:15:40.699 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-296352743437 2024-09-10T20:15:40.700 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:15:40.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:15:40.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-296352743437 2024-09-10T20:15:40.702 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:15:40.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=296352743437 2024-09-10T20:15:40.703 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 296352743437 2024-09-10T20:15:40.704 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 296352743437' 2024-09-10T20:15:40.704 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:15:41.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 296352743437 -lt 296352743437 2024-09-10T20:15:41.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:15:41.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:41.192 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:41.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 32 == 0 2024-09-10T20:15:41.796 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:41.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:41.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:41.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:41.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:41.797 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:41.798 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:42.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:42.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:42.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:42.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:42.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:42.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:15:42.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:15:42.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:15:42.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:42.875 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:15:42.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:42.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:42.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:42.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:42.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:42.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:42.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:42.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:43.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:43.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:43.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:43.421 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:44.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:44.010 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:44.011 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:44.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:44.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:44.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:15:44.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:44.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:15:44.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:44.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:44.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:44.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:44.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:44.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:44.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:44.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:45.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:45.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:45.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:45.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:45.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:45.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:45.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:45.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:45.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:45.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:45.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:45.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:45.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:15:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:46.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:15:46.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:46.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:46.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:46.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:46.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:46.887 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:46.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:46.888 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:47.344 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:47.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:47.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:47.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:47.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:47.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:47.940 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:15:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:48.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:15:49.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:49.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:49.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:49.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:49.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:49.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:49.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:49.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:49.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:49.826 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:49.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:49.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:50.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:50.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:50.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:50.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:50.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:50.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:50.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:50.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:50.410 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:50.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:50.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:50.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:15:50.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:50.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:15:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:52.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:52.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:52.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:52.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:52.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:52.590 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:53.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:53.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:53.043 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:53.044 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:53.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:53.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:53.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:53.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:53.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:53.638 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:54.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:54.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:54.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:15:54.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:54.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:15:57.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:15:57.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:15:57.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:15:57.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:15:57.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:15:57.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:15:57.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:15:57.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:15:57.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:15:57.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:15:57.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:15:57.899 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:15:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:15:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:15:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:15:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:15:58.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:15:58.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:15:58.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:15:58.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:15:58.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:15:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:15:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:15:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:15:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:15:59.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:16:05.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:16:05.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:16:05.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:16:05.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:16:05.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:16:05.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:16:05.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:16:05.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:16:05.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:16:05.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:16:05.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:16:05.973 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:16:06.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:16:06.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:16:06.555 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:16:07.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:16:07.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:16:07.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:16:07.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:16:07.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:16:14.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:16:14.466 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:16:14.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:16:14.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:16:14.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:16:14.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:16:14.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:16:14.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:16:14.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:16:14.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:16:14.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:16:14.931 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:16:15.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 32 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:16:15.527 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:16:16.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:16:16.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:16:16.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:16:16.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:16:16.721 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:16:16.721 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "8a3964a9-1551-4cea-95e3-4290ddd16e3b", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:16:16.721212+0000", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:16:16.722 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:16:16.723 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 64 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.724 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.2 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.3 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.4 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.725 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.5 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.6 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.7 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.8 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.726 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.9 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.b is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.c is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.d is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.727 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.e is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.f is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.10 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.11 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.728 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.12 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.13 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.14 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.15 is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.729 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.16 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.17 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.18 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.19 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.730 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1a is stuck undersized for 74s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1b is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1c is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1d is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.731 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1e is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.1f is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,0]" 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:16:16.732 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:16:16.733 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:12:01.404918Z", 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:12:01.404918Z", 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:16:16.734 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:16:16.735 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:16:16.736 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:16:16.737 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 78, 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:12:02.326494+0000", 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:15:01.909107+0000", 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:14:57.577984+0000", 2024-09-10T20:16:16.738 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:15:00.108037+0000", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 9, 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:16:16.739 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:13:46.168014+0000", 2024-09-10T20:16:16.740 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 1, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool", 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 0, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "on", 2024-09-10T20:16:16.741 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 32, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 21, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 32, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 32, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 32, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "74", 2024-09-10T20:16:16.742 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "33", 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 0, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 0, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [], 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:16:16.743 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "", 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.744 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {}, 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "read_balance": { 2024-09-10T20:16:16.745 INFO:tasks.workunit.client.0.smithi039.stdout: "score_type": "Fair distribution", 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "score_acting": 1.7799999713897705, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "score_stable": 1.7799999713897705, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_score": 0.67000001668930054, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_acting": 1.190000057220459, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "raw_score_stable": 1.190000057220459, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity": 1, 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:16:16.746 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "11e1e427-b96e-40c4-8c63-b5bfbbef0271", 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 47, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 64, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 74, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 55, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.747 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:16:16.748 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:16:16.749 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.750 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1056624563 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.751 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1056624563", 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1056624563", 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1056624563", 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1056624563", 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "f6d81b4e-47a0-4251-8ee2-fe937090de25", 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:16:16.752 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 52, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 56, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.753 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.754 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.755 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.756 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3965607747 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3965607747", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3965607747", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3965607747", 2024-09-10T20:16:16.757 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3965607747", 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "0205c44a-8881-450b-ac91-9a38b8066f16", 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-10T20:16:16.758 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 47, 2024-09-10T20:16:16.759 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 69, 2024-09-10T20:16:16.759 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 76, 2024-09-10T20:16:16.759 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 57, 2024-09-10T20:16:16.759 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:16:16.759 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:16:16.759 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.760 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.761 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.762 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:16:16.763 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3778029559 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/3778029559", 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/3778029559", 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3778029559", 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3778029559", 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.764 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:14:44.889964+0000", 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:12:34.901905+0000", 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.765 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:14:45.486363+0000", 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:13:07.680417+0000", 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:14:46.048103+0000", 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:16:16.766 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:13:42.255582+0000", 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:16:16.767 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:16:16.768 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1056624563,v1:127.0.0.1:6805/1056624563]", 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:16:16.769 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:16:16.770 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:12:08.744619Z", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:16:16.771 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1056624563,v1:127.0.0.1:6803/1056624563]", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1056624563,v1:127.0.0.1:6809/1056624563]", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1056624563,v1:127.0.0.1:6807/1056624563]", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:16:16.772 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:16:16.773 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3965607747,v1:127.0.0.1:6813/3965607747]", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:16:16.774 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:12:41.572217Z", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:16:16.775 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3965607747,v1:127.0.0.1:6811/3965607747]", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3965607747,v1:127.0.0.1:6817/3965607747]", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3965607747,v1:127.0.0.1:6815/3965607747]", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:16:16.776 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:16:16.777 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/3778029559,v1:127.0.0.1:6813/3778029559]", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:16:16.778 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:13:15.126433Z", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:16:16.779 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/3778029559,v1:127.0.0.1:6811/3778029559]", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3778029559,v1:127.0.0.1:6817/3778029559]", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3778029559,v1:127.0.0.1:6815/3778029559]", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:16:16.780 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [ 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:16:16.781 INFO:tasks.workunit.client.0.smithi039.stdout: "floor": 36 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 65 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:16:16.782 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 69 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stderr:report 1678658510 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 78, 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:16:16.783 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:16:16.784 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.785 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:16:16.786 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.787 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.788 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.789 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:16:16.790 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.791 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:16:16.792 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:16:16.793 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.794 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.795 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:16:16.796 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 0, 2024-09-10T20:16:16.797 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:12:02:302432+0000", 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:16:16.798 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:16:16.799 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:16:16.800 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 32, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62532, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:16:16.801 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 14, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 14, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:16:16.802 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:16:16.803 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:16:16.804 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 524, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 524, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-10T20:16:16.805 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84464, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2888, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488336, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036056064, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2957312, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 752360, 2024-09-10T20:16:16.806 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.807 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 97, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 97, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 97000000, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 97000000 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 96, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:16:16.808 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 84464, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2888, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314488336, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036056064, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:16:16.810 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2957312, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 752360, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:16:16.811 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 97, 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 97, 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 97000000, 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 97000000 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.812 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 32, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62532, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 0, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:16:16.813 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 14, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 14, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:16:16.814 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:16:16.815 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1220608, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 185266, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 524, 2024-09-10T20:16:16.816 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 524, 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 64, 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 64, 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 274877906965 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672989 2024-09-10T20:16:16.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 296352743444 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 32 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.818 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 19, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 13, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 32, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:16:16.819 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 7, 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:15:58.280575+0000", 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-10T20:16:16.820 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 116, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 497, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 675, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:16:16.821 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:16:16.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:16:16.822 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-10T20:16:16.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-10T20:16:16.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.367576+0000 34'58 78:157 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:52:02.710344+0000 3 0 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368058+0000 42'59 78:159 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:06:02.950434+0000 2 0 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.369421+0000 34'59 78:138 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:50:21.167484+0000 4 0 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.266233+0000 41'58 78:143 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:51:38.070635+0000 3 0 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:02.061489+0000 32'57 78:128 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:55.584648+0000 3 0 2024-09-10T20:16:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.061357+0000 40'59 78:138 [2,0] 2 [2,0] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:17:05.827200+0000 4 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 3 0 0 1851 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.488026+0000 41'60 78:157 [2,0] 2 [2,0] 2 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:24:22.645126+0000 2 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.977085+0000 32'57 78:139 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:23:09.257405+0000 2 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2468 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.364911+0000 36'60 78:166 [0,2] 0 [0,2] 0 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:30:24.359258+0000 4 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.487584+0000 40'60 78:141 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:25:11.278817+0000 5 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362640+0000 34'58 78:130 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:53:02.440421+0000 2 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.370354+0000 42'59 78:149 [2,0] 2 [2,0] 2 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:03:02.756244+0000 5 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.364829+0000 35'58 78:146 [0,2] 0 [0,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:57:36.818603+0000 1 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.365134+0000 39'58 78:137 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:11:45.912858+0000 2 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488077+0000 38'59 78:134 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:19:02.438290+0000 0 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362517+0000 32'57 78:134 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:08:51.120229+0000 2 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.363402+0000 41'58 78:142 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:40:34.279604+0000 2 0 2024-09-10T20:16:17.161 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368616+0000 40'59 78:144 [2,0] 2 [2,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:58:38.628580+0000 3 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 4 0 0 2818 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.368558+0000 78'60 78:160 [2,0] 2 [2,0] 2 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:31:13.413252+0000 4 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.369057+0000 34'58 78:151 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:52:04.039387+0000 4 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362770+0000 41'58 78:151 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:10:08.517960+0000 3 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.365481+0000 32'57 78:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:07:31.456366+0000 3 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362099+0000 32'57 78:142 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:21:18.619051+0000 1 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365931+0000 41'59 78:155 [0,2] 0 [0,2] 0 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:41:22.588379+0000 3 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488142+0000 41'59 78:147 [2,0] 2 [2,0] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:15:07.508816+0000 3 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.413696+0000 34'58 78:143 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:45:39.663228+0000 3 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.362516+0000 34'59 78:155 [0,2] 0 [0,2] 0 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:58:06.455346+0000 4 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.369618+0000 32'57 78:156 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:33:43.963351+0000 1 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2333 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368029+0000 41'59 78:156 [2,0] 2 [2,0] 2 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 user requested scrub scheduled @ 2024-09-10T20:16:02.201477+0000 0 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.367551+0000 32'57 78:132 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:56:48.695621+0000 2 0 2024-09-10T20:16:17.162 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.061528+0000 38'58 78:138 [2,0] 2 [2,0] 2 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:37:36.538557+0000 4 0 2024-09-10T20:16:17.163 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365641+0000 41'59 78:156 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:20.398497+0000 4 0 2024-09-10T20:16:17.163 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:17.163 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:17.163 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:17.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:16:17.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-10T20:16:17.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:16:17.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:16:17.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:16:17.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:16:17.177 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:16:17.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:16:17.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-10T20:16:17.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok set_recovery_delay 100000 2024-09-10T20:16:17.281 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:16:17.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-10T20:16:17.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-10T20:16:17.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:16:17.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:16:17.296 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:16:17.296 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:16:17.296 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:16:17.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:16:17.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-10T20:16:17.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok set_recovery_delay 100000 2024-09-10T20:16:17.398 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:16:17.409 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-10T20:16:17.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-10T20:16:17.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:17.832 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:17.832 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.367576+0000 34'58 78:157 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:52:02.710344+0000 3 0 2024-09-10T20:16:17.832 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368058+0000 42'59 78:159 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:06:02.950434+0000 2 0 2024-09-10T20:16:17.832 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.369421+0000 34'59 78:138 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:50:21.167484+0000 4 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.266233+0000 41'58 78:143 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:51:38.070635+0000 3 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:02.061489+0000 32'57 78:128 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:55.584648+0000 3 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.061357+0000 40'59 78:138 [2,0] 2 [2,0] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:17:05.827200+0000 4 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 3 0 0 1851 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.488026+0000 41'60 78:157 [2,0] 2 [2,0] 2 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:24:22.645126+0000 2 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.977085+0000 32'57 78:139 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:23:09.257405+0000 2 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2468 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.364911+0000 36'60 78:166 [0,2] 0 [0,2] 0 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:30:24.359258+0000 4 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.487584+0000 40'60 78:141 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:25:11.278817+0000 5 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362640+0000 34'58 78:130 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:53:02.440421+0000 2 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.370354+0000 42'59 78:149 [2,0] 2 [2,0] 2 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:03:02.756244+0000 5 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.364829+0000 35'58 78:146 [0,2] 0 [0,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:57:36.818603+0000 1 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.365134+0000 39'58 78:137 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:11:45.912858+0000 2 0 2024-09-10T20:16:17.833 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488077+0000 38'59 78:134 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:19:02.438290+0000 0 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362517+0000 32'57 78:134 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:08:51.120229+0000 2 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.363402+0000 41'58 78:142 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:40:34.279604+0000 2 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368616+0000 40'59 78:144 [2,0] 2 [2,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:58:38.628580+0000 3 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 4 0 0 2818 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.368558+0000 78'60 78:160 [2,0] 2 [2,0] 2 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:31:13.413252+0000 4 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.369057+0000 34'58 78:151 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:52:04.039387+0000 4 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362770+0000 41'58 78:151 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:10:08.517960+0000 3 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.365481+0000 32'57 78:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:07:31.456366+0000 3 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362099+0000 32'57 78:142 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:21:18.619051+0000 1 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365931+0000 41'59 78:155 [0,2] 0 [0,2] 0 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:41:22.588379+0000 3 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488142+0000 41'59 78:147 [2,0] 2 [2,0] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:15:07.508816+0000 3 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.413696+0000 34'58 78:143 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:45:39.663228+0000 3 0 2024-09-10T20:16:17.834 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.362516+0000 34'59 78:155 [0,2] 0 [0,2] 0 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:58:06.455346+0000 4 0 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.369618+0000 32'57 78:156 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:33:43.963351+0000 1 0 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2333 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368029+0000 41'59 78:156 [2,0] 2 [2,0] 2 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 user requested scrub scheduled @ 2024-09-10T20:16:02.201477+0000 0 0 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.367551+0000 32'57 78:132 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:56:48.695621+0000 2 0 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.061528+0000 38'58 78:138 [2,0] 2 [2,0] 2 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:37:36.538557+0000 4 0 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365641+0000 41'59 78:156 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:20.398497+0000 4 0 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:17.835 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:17.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-10T20:16:18.538 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-10T20:16:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-10T20:16:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:16:18.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:16:18.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:16:18.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:16:18.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:16:18.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:16:18.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:16:18.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:16:18.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:16:18.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:16:18.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:16:18.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:16:18.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:16:18.569 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:16:18.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:16:18.607 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:18.607+0000 7f3d57bae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:16:18.609 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:18.611+0000 7f3d57bae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:16:18.611 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:18.611+0000 7f3d57bae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:16:19.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-10T20:16:19.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-10T20:16:19.884 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:19.883+0000 7f3d57bae7c0 -1 Falling back to public interface 2024-09-10T20:16:22.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:21.999+0000 7f3d57bae7c0 -1 osd.1 53 log_to_monitors true 2024-09-10T20:16:24.038 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-10T20:16:24.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-10T20:16:24.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.367576+0000 34'58 80:161 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:52:02.710344+0000 3 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368058+0000 42'59 80:163 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:06:02.950434+0000 2 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.369421+0000 34'59 80:142 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:50:21.167484+0000 4 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.266233+0000 41'58 80:146 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:51:38.070635+0000 3 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:02.061489+0000 32'57 80:132 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:55.584648+0000 3 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.061357+0000 40'59 80:142 [2,0] 2 [2,0] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:17:05.827200+0000 4 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 3 0 0 1851 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.488026+0000 41'60 80:161 [2,0] 2 [2,0] 2 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:24:22.645126+0000 2 0 2024-09-10T20:16:24.476 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.977085+0000 32'57 80:143 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:23:09.257405+0000 2 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2468 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.364911+0000 36'60 80:169 [0,2] 0 [0,2] 0 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:30:24.359258+0000 4 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.487584+0000 40'60 80:145 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:25:11.278817+0000 5 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362640+0000 34'58 80:133 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:53:02.440421+0000 2 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.370354+0000 42'59 80:153 [2,0] 2 [2,0] 2 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:03:02.756244+0000 5 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.364829+0000 35'58 80:149 [0,2] 0 [0,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:57:36.818603+0000 1 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.365134+0000 39'58 80:140 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:11:45.912858+0000 2 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488077+0000 38'59 80:138 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:19:02.438290+0000 0 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362517+0000 32'57 80:137 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:08:51.120229+0000 2 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.363402+0000 41'58 80:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:40:34.279604+0000 2 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368616+0000 40'59 80:148 [2,0] 2 [2,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:58:38.628580+0000 3 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 4 0 0 2818 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.368558+0000 78'60 80:164 [2,0] 2 [2,0] 2 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:31:13.413252+0000 4 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.369057+0000 34'58 80:155 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:52:04.039387+0000 4 0 2024-09-10T20:16:24.477 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362770+0000 41'58 80:154 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:10:08.517960+0000 3 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.365481+0000 32'57 80:148 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:07:31.456366+0000 3 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362099+0000 32'57 80:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:21:18.619051+0000 1 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365931+0000 41'59 80:158 [0,2] 0 [0,2] 0 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:41:22.588379+0000 3 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488142+0000 41'59 80:151 [2,0] 2 [2,0] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:15:07.508816+0000 3 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.413696+0000 34'58 80:147 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:45:39.663228+0000 3 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.362516+0000 34'59 80:158 [0,2] 0 [0,2] 0 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:58:06.455346+0000 4 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.369618+0000 32'57 80:160 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:33:43.963351+0000 1 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2333 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368029+0000 41'59 80:160 [2,0] 2 [2,0] 2 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 user requested scrub scheduled @ 2024-09-10T20:17:02.395655+0000 0 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.367551+0000 32'57 80:136 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:56:48.695621+0000 2 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.061528+0000 38'58 80:142 [2,0] 2 [2,0] 2 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:37:36.538557+0000 4 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365641+0000 41'59 80:159 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:20.398497+0000 4 0 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:24.478 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:24.479 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:24.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-10T20:16:24.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:16:24.492 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:16:24.492 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:16:24.493 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:16:24.493 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:16:24.493 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:16:24.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-10T20:16:24.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok set_recovery_delay 100000 2024-09-10T20:16:24.609 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:16:24.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-10T20:16:25.276 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-10T20:16:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-10T20:16:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:16:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:16:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:16:25.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:16:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:16:25.299 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:16:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:16:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:16:25.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:16:26.224 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 64 up_thru 83 down_at 55 last_clean_interval [5,47) [v2:127.0.0.1:6802/1056624563,v1:127.0.0.1:6803/1056624563] [v2:127.0.0.1:6804/1056624563,v1:127.0.0.1:6805/1056624563] exists,up 11e1e427-b96e-40c4-8c63-b5bfbbef0271 2024-09-10T20:16:26.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:16:26.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:16:26.226 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:16:26.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:16:26.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:16:26.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:16:26.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:16:26.831 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 56 last_clean_interval [10,53) [v2:127.0.0.1:6818/3724677151,v1:127.0.0.1:6819/3724677151] [v2:127.0.0.1:6820/3724677151,v1:127.0.0.1:6821/3724677151] exists,up f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:16:26.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:16:26.834 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:16:26.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:16:26.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:16:27.291 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 69 up_thru 83 down_at 57 last_clean_interval [15,47) [v2:127.0.0.1:6810/3778029559,v1:127.0.0.1:6811/3778029559] [v2:127.0.0.1:6812/3778029559,v1:127.0.0.1:6813/3778029559] exists,up 0205c44a-8881-450b-ac91-9a38b8066f16 2024-09-10T20:16:27.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:16:27.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:16:27.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:16:27.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:27.719 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:27.719 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.367576+0000 34'58 80:161 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:52:02.710344+0000 3 0 2024-09-10T20:16:27.719 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368058+0000 42'59 80:163 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:06:02.950434+0000 2 0 2024-09-10T20:16:27.719 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.369421+0000 34'59 80:142 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:50:21.167484+0000 4 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.266233+0000 41'58 80:146 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:51:38.070635+0000 3 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:02.061489+0000 32'57 80:132 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:55.584648+0000 3 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.061357+0000 40'59 80:142 [2,0] 2 [2,0] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:17:05.827200+0000 4 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 3 0 0 1851 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.488026+0000 41'60 80:161 [2,0] 2 [2,0] 2 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:24:22.645126+0000 2 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.977085+0000 32'57 80:143 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:23:09.257405+0000 2 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2468 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.364911+0000 36'60 80:169 [0,2] 0 [0,2] 0 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:30:24.359258+0000 4 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.487584+0000 40'60 80:145 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:25:11.278817+0000 5 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362640+0000 34'58 80:133 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:53:02.440421+0000 2 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.370354+0000 42'59 80:153 [2,0] 2 [2,0] 2 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:03:02.756244+0000 5 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.364829+0000 35'58 80:149 [0,2] 0 [0,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:57:36.818603+0000 1 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.365134+0000 39'58 80:140 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:11:45.912858+0000 2 0 2024-09-10T20:16:27.720 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488077+0000 38'59 80:138 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:19:02.438290+0000 0 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362517+0000 32'57 80:137 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:08:51.120229+0000 2 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.363402+0000 41'58 80:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:40:34.279604+0000 2 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368616+0000 40'59 80:148 [2,0] 2 [2,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:58:38.628580+0000 3 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 4 0 0 2818 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.368558+0000 78'60 80:164 [2,0] 2 [2,0] 2 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:31:13.413252+0000 4 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.369057+0000 34'58 80:155 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:52:04.039387+0000 4 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362770+0000 41'58 80:154 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:10:08.517960+0000 3 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.365481+0000 32'57 80:148 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:07:31.456366+0000 3 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362099+0000 32'57 80:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:21:18.619051+0000 1 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365931+0000 41'59 80:158 [0,2] 0 [0,2] 0 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:41:22.588379+0000 3 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488142+0000 41'59 80:151 [2,0] 2 [2,0] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:15:07.508816+0000 3 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.413696+0000 34'58 80:147 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:45:39.663228+0000 3 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.362516+0000 34'59 80:158 [0,2] 0 [0,2] 0 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:58:06.455346+0000 4 0 2024-09-10T20:16:27.721 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.369618+0000 32'57 80:160 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:33:43.963351+0000 1 0 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2333 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368029+0000 41'59 80:160 [2,0] 2 [2,0] 2 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 user requested scrub scheduled @ 2024-09-10T20:17:02.395655+0000 0 0 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.367551+0000 32'57 80:136 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:56:48.695621+0000 2 0 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.061528+0000 38'58 80:142 [2,0] 2 [2,0] 2 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:37:36.538557+0000 4 0 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365641+0000 41'59 80:159 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:20.398497+0000 4 0 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:27.722 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:27.736 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-10T20:16:27.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-10T20:16:27.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:28.170 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:28.171 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.367576+0000 34'58 80:161 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:52:02.710344+0000 3 0 2024-09-10T20:16:28.171 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368058+0000 42'59 80:163 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:06:02.950434+0000 2 0 2024-09-10T20:16:28.171 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.369421+0000 34'59 80:142 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:50:21.167484+0000 4 0 2024-09-10T20:16:28.171 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.266233+0000 41'58 80:146 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:51:38.070635+0000 3 0 2024-09-10T20:16:28.171 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:02.061489+0000 32'57 80:132 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:19.124048+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:55.584648+0000 3 0 2024-09-10T20:16:28.172 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.061357+0000 40'59 80:142 [2,0] 2 [2,0] 2 37'58 2024-09-10T20:14:19.358328+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:17:05.827200+0000 4 0 2024-09-10T20:16:28.172 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 3 0 0 1851 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.488026+0000 41'60 80:161 [2,0] 2 [2,0] 2 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:24:22.645126+0000 2 0 2024-09-10T20:16:28.172 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.977085+0000 32'57 80:143 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:23:09.257405+0000 2 0 2024-09-10T20:16:28.172 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 4 0 0 2468 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.364911+0000 36'60 80:169 [0,2] 0 [0,2] 0 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:30:24.359258+0000 4 0 2024-09-10T20:16:28.172 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 6 0 0 3702 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:02.487584+0000 40'60 80:145 [2,0] 2 [2,0] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:25:11.278817+0000 5 0 2024-09-10T20:16:28.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 2 0 0 1234 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362640+0000 34'58 80:133 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:53:02.440421+0000 2 0 2024-09-10T20:16:28.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.370354+0000 42'59 80:153 [2,0] 2 [2,0] 2 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:03:02.756244+0000 5 0 2024-09-10T20:16:28.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 1 0 0 617 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.364829+0000 35'58 80:149 [0,2] 0 [0,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:57:36.818603+0000 1 0 2024-09-10T20:16:28.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.365134+0000 39'58 80:140 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:11:45.912858+0000 2 0 2024-09-10T20:16:28.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 2 0 0 1234 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488077+0000 38'59 80:138 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:19:02.438290+0000 0 0 2024-09-10T20:16:28.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362517+0000 32'57 80:137 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:08:51.120229+0000 2 0 2024-09-10T20:16:28.174 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.363402+0000 41'58 80:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:40:34.279604+0000 2 0 2024-09-10T20:16:28.174 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368616+0000 40'59 80:148 [2,0] 2 [2,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:58:38.628580+0000 3 0 2024-09-10T20:16:28.174 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 0 4 0 0 2818 0 0 18 42 18 active+undersized+degraded 2024-09-10T20:15:01.368558+0000 78'60 80:164 [2,0] 2 [2,0] 2 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:31:13.413252+0000 4 0 2024-09-10T20:16:28.174 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.369057+0000 34'58 80:155 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:52:04.039387+0000 4 0 2024-09-10T20:16:28.174 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:01.362770+0000 41'58 80:154 [0,2] 0 [0,2] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:10:08.517960+0000 3 0 2024-09-10T20:16:28.174 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 3 0 0 1851 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.365481+0000 32'57 80:148 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:07:31.456366+0000 3 0 2024-09-10T20:16:28.175 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.362099+0000 32'57 80:145 [0,2] 0 [0,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:21:18.619051+0000 1 0 2024-09-10T20:16:28.175 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365931+0000 41'59 80:158 [0,2] 0 [0,2] 0 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:41:22.588379+0000 3 0 2024-09-10T20:16:28.175 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 3 0 0 1851 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:02.488142+0000 41'59 80:151 [2,0] 2 [2,0] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:15:07.508816+0000 3 0 2024-09-10T20:16:28.175 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 3 0 0 1851 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.413696+0000 34'58 80:147 [2,0] 2 [2,0] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:45:39.663228+0000 3 0 2024-09-10T20:16:28.175 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 4 0 0 2468 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.362516+0000 34'59 80:158 [0,2] 0 [0,2] 0 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:58:06.455346+0000 4 0 2024-09-10T20:16:28.175 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 1 0 0 617 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.369618+0000 32'57 80:160 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:33:43.963351+0000 1 0 2024-09-10T20:16:28.176 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 4 0 4 0 0 2333 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.368029+0000 41'59 80:160 [2,0] 2 [2,0] 2 0'0 2024-09-10T20:13:46.218219+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 user requested scrub scheduled @ 2024-09-10T20:17:02.395655+0000 0 0 2024-09-10T20:16:28.176 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 2 0 0 1234 0 0 15 42 15 active+undersized+degraded 2024-09-10T20:15:01.367551+0000 32'57 80:136 [2,0] 2 [2,0] 2 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:56:48.695621+0000 2 0 2024-09-10T20:16:28.176 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 4 0 0 2468 0 0 16 42 16 active+undersized+degraded 2024-09-10T20:15:02.061528+0000 38'58 80:142 [2,0] 2 [2,0] 2 38'58 2024-09-10T20:14:24.060125+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T23:37:36.538557+0000 4 0 2024-09-10T20:16:28.176 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 0 5 0 0 3085 0 0 17 42 17 active+undersized+degraded 2024-09-10T20:15:01.365641+0000 41'59 80:159 [0,2] 0 [0,2] 0 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:20.398497+0000 4 0 2024-09-10T20:16:28.176 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:28.176 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:28.177 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:28.186 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-10T20:16:28.292 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-10T20:16:28.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-10T20:16:28.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:28.751 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:28.751 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.240428+0000 34'58 85:177 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:24:13.819425+0000 3 0 2024-09-10T20:16:28.752 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.312565+0000 42'59 85:162 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:34:35.329404+0000 2 0 2024-09-10T20:16:28.752 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:26.236434+0000 34'59 85:158 [2,1,0] 2 [2,1,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:57:07.048780+0000 4 0 2024-09-10T20:16:28.752 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.314391+0000 41'58 85:177 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:19:10.669876+0000 3 0 2024-09-10T20:16:28.752 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:16:26.260749+0000 32'57 85:150 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:37.778164+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:26:17.886534+0000 3 0 2024-09-10T20:16:28.752 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 1 1 0 0 3085 0 0 17 42 17 active+recovery_wait+degraded 2024-09-10T20:16:26.310666+0000 40'59 85:176 [1,2,0] 1 [1,2,0] 1 47'60 2024-09-10T20:14:35.710973+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 active+clean 2024-09-10T20:16:26.312925+0000 41'60 85:165 [1,2,0] 1 [1,2,0] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:43:38.981727+0000 2 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:26.314900+0000 32'57 85:142 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:12:54.203987+0000 2 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2468 0 0 18 42 18 active+clean 2024-09-10T20:16:26.311426+0000 36'60 85:163 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:40:34.233411+0000 4 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 active+clean 2024-09-10T20:16:26.236499+0000 40'60 85:161 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:12:10.014880+0000 5 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:16:26.312172+0000 34'58 85:127 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:34:09.822188+0000 2 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:16:26.312548+0000 42'59 85:148 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:40:27.519941+0000 5 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 active+clean 2024-09-10T20:16:26.563098+0000 35'58 85:165 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:16:49.233898+0000 1 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.311073+0000 39'58 85:136 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:46:44.723565+0000 2 0 2024-09-10T20:16:28.753 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 active+clean 2024-09-10T20:16:26.240387+0000 38'59 85:154 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:48:43.885594+0000 0 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:26.313362+0000 32'57 85:131 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:57:06.818798+0000 2 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.312913+0000 41'58 85:141 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:01:45.603422+0000 2 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.237451+0000 40'59 85:164 [2,1,0] 2 [2,1,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:51:03.393761+0000 3 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 1 1 0 0 2818 0 0 18 42 18 active+recovery_wait+degraded 2024-09-10T20:16:26.311954+0000 78'60 85:165 [1,2,0] 1 [1,2,0] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 no scrub is scheduled 4 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:26.236519+0000 34'58 85:171 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:21.691135+0000 4 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.565145+0000 41'58 85:170 [0,2,1] 0 [0,2,1] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:32:24.058946+0000 3 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1851 0 0 15 42 15 active+recovery_wait+degraded 2024-09-10T20:16:26.561889+0000 32'57 85:162 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 3 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:26.563445+0000 32'57 85:161 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:16:28.258638+0000 1 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.168441+0000 41'59 85:152 [1,0,2] 1 [1,0,2] 1 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:21:24.503778+0000 3 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.240180+0000 41'59 85:167 [2,0,1] 2 [2,0,1] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:38:36.670537+0000 3 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.239645+0000 34'58 85:163 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:59:55.849708+0000 3 0 2024-09-10T20:16:28.754 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:26.315115+0000 34'59 85:152 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:32:37.938192+0000 4 0 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:26.311799+0000 32'57 85:155 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:26:14.580641+0000 1 0 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:27.009459+0000 41'59 85:179 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:16:27.009391+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:09:17.192342+0000 3 0 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:26.313252+0000 32'57 85:140 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:12:08.830045+0000 2 0 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:26.311353+0000 38'58 85:163 [1,2,0] 1 [1,2,0] 1 38'58 2024-09-10T20:14:38.741886+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:13:37.331896+0000 4 0 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 3085 0 0 17 42 17 active+recovering+degraded 2024-09-10T20:16:26.421110+0000 41'59 85:191 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:28.755 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:28.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-10T20:16:28.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:16:28.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:16:28.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:16:28.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:16:28.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.27146.out 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27146.out 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-10T20:16:28.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27146.out 2024-09-10T20:16:30.437 INFO:tasks.workunit.client.0.smithi039.stderr:Exporting 1.1b info 1.1b( v 41'59 lc 0'0 (31'42,41'59] local-lis/les=83/84 n=5 ec=33/23 lis/c=83/52 les/c/f=84/53/0 sis=83) 2024-09-10T20:16:30.445 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-10T20:16:30.455 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-10T20:16:30.473 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-10T20:16:30.496 INFO:tasks.workunit.client.0.smithi039.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-10T20:16:30.511 INFO:tasks.workunit.client.0.smithi039.stderr:Export successful 2024-09-10T20:16:30.512 INFO:tasks.workunit.client.0.smithi039.stdout: marking collection for removal 2024-09-10T20:16:30.512 INFO:tasks.workunit.client.0.smithi039.stdout:setting '_remove' omap key 2024-09-10T20:16:30.512 INFO:tasks.workunit.client.0.smithi039.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-10T20:16:30.654 INFO:tasks.workunit.client.0.smithi039.stderr:Remove successful 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27146.out 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-10T20:16:31.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27146.out 2024-09-10T20:16:32.235 INFO:tasks.workunit.client.0.smithi039.stdout:get_pg_num_history pg_num_history pg_num_history(e85 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-09-10T20:16:32.235 INFO:tasks.workunit.client.0.smithi039.stdout:Importing pgid 1.1b 2024-09-10T20:16:32.235 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-10T20:16:32.235 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:16:32.298 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-10T20:16:32.298 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:16:32.332 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-10T20:16:32.332 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:16:32.375 INFO:tasks.workunit.client.0.smithi039.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-10T20:16:32.375 INFO:tasks.workunit.client.0.smithi039.stdout:snapset 0=[]:{} 2024-09-10T20:16:32.432 INFO:tasks.workunit.client.0.smithi039.stdout:write_pg epoch 85 info 1.1b( v 41'59 lc 0'0 (31'42,41'59] local-lis/les=83/84 n=5 ec=33/23 lis/c=83/52 les/c/f=84/53/0 sis=83) 2024-09-10T20:16:32.482 INFO:tasks.workunit.client.0.smithi039.stdout:Import successful 2024-09-10T20:16:33.019 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-10T20:16:33.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-10T20:16:33.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:33.467 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:33.467 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.240428+0000 34'58 85:177 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:24:13.819425+0000 3 0 2024-09-10T20:16:33.467 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.312565+0000 42'59 85:162 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:34:35.329404+0000 2 0 2024-09-10T20:16:33.467 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:26.236434+0000 34'59 85:158 [2,1,0] 2 [2,1,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:57:07.048780+0000 4 0 2024-09-10T20:16:33.467 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.314391+0000 41'58 85:177 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:19:10.669876+0000 3 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:16:26.260749+0000 32'57 85:150 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:37.778164+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T22:26:17.886534+0000 3 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 1 1 0 0 3085 0 0 17 42 17 active+recovery_wait+degraded 2024-09-10T20:16:26.310666+0000 40'59 85:176 [1,2,0] 1 [1,2,0] 1 47'60 2024-09-10T20:14:35.710973+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 active+clean 2024-09-10T20:16:26.312925+0000 41'60 85:165 [1,2,0] 1 [1,2,0] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:43:38.981727+0000 2 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:26.314900+0000 32'57 85:142 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:12:54.203987+0000 2 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 4 0 0 0 0 2468 0 0 18 42 18 active+clean 2024-09-10T20:16:26.311426+0000 36'60 85:163 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:40:34.233411+0000 4 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 active+clean 2024-09-10T20:16:26.236499+0000 40'60 85:161 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:12:10.014880+0000 5 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:16:26.312172+0000 34'58 85:127 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:34:09.822188+0000 2 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:16:26.312548+0000 42'59 85:148 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:40:27.519941+0000 5 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 active+clean 2024-09-10T20:16:26.563098+0000 35'58 85:165 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:16:49.233898+0000 1 0 2024-09-10T20:16:33.468 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.311073+0000 39'58 85:136 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:46:44.723565+0000 2 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 active+clean 2024-09-10T20:16:26.240387+0000 38'59 85:154 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:48:43.885594+0000 0 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:26.313362+0000 32'57 85:131 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:57:06.818798+0000 2 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.312913+0000 41'58 85:141 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:01:45.603422+0000 2 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.237451+0000 40'59 85:164 [2,1,0] 2 [2,1,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:51:03.393761+0000 3 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 1 1 0 0 2818 0 0 18 42 18 active+recovery_wait+degraded 2024-09-10T20:16:26.311954+0000 78'60 85:165 [1,2,0] 1 [1,2,0] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 no scrub is scheduled 4 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:26.236519+0000 34'58 85:171 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:50:21.691135+0000 4 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.565145+0000 41'58 85:170 [0,2,1] 0 [0,2,1] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:32:24.058946+0000 3 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1851 0 0 15 42 15 active+recovery_wait+degraded 2024-09-10T20:16:26.561889+0000 32'57 85:162 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 3 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:26.563445+0000 32'57 85:161 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:16:28.258638+0000 1 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.168441+0000 41'59 85:152 [1,0,2] 1 [1,0,2] 1 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:21:24.503778+0000 3 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:26.240180+0000 41'59 85:167 [2,0,1] 2 [2,0,1] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T02:38:36.670537+0000 3 0 2024-09-10T20:16:33.469 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:26.239645+0000 34'58 85:163 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:59:55.849708+0000 3 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:26.315115+0000 34'59 85:152 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:32:37.938192+0000 4 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:26.311799+0000 32'57 85:155 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T05:26:14.580641+0000 1 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:27.009459+0000 41'59 85:179 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:16:27.009391+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:09:17.192342+0000 3 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:26.313252+0000 32'57 85:140 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:12:08.830045+0000 2 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:26.311353+0000 38'58 85:163 [1,2,0] 1 [1,2,0] 1 38'58 2024-09-10T20:14:38.741886+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:13:37.331896+0000 4 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 3085 0 0 17 42 17 active+recovering+degraded 2024-09-10T20:16:26.421110+0000 41'59 85:191 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:33.470 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:16:33.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:16:33.487 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:16:33.488 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:16:33.488 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:16:33.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:16:33.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:16:33.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:16:33.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:16:33.490 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:16:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:16:33.524 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:33.523+0000 7f62e8ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:16:33.526 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:33.527+0000 7f62e8ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:16:33.527 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:33.527+0000 7f62e8ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:16:33.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:16:33.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:16:33.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:16:33.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:16:33.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:16:33.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:16:33.991 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:16:33.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:16:33.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:16:33.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:16:34.496 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 56 last_clean_interval [10,53) [v2:127.0.0.1:6818/3724677151,v1:127.0.0.1:6819/3724677151] [v2:127.0.0.1:6820/3724677151,v1:127.0.0.1:6821/3724677151] exists,up f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:16:34.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:16:34.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:16:34.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:16:34.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-10T20:16:34.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:16:34.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:16:34.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:16:34.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:16:34.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:16:34.498 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:16:34.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:16:34.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:16:34.499 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:16:34.788 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:34.787+0000 7f62e8ec47c0 -1 Falling back to public interface 2024-09-10T20:16:34.991 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 83 up_thru 83 down_at 56 last_clean_interval [10,53) [v2:127.0.0.1:6818/3724677151,v1:127.0.0.1:6819/3724677151] [v2:127.0.0.1:6820/3724677151,v1:127.0.0.1:6821/3724677151] exists,up f6d81b4e-47a0-4251-8ee2-fe937090de25 2024-09-10T20:16:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:16:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:16:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:16:34.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-10T20:16:36.523 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:16:36.523+0000 7f62e8ec47c0 -1 osd.1 85 log_to_monitors true 2024-09-10T20:16:54.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-10T20:16:54.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:16:54.997 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:16:54.997 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:16:54.998 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:16:54.998 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:16:54.998 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:16:54.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-10T20:16:54.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok dump_ops_in_flight 2024-09-10T20:16:55.115 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:16:55.115 INFO:tasks.workunit.client.0.smithi039.stdout: "ops": [], 2024-09-10T20:16:55.115 INFO:tasks.workunit.client.0.smithi039.stdout: "num_ops": 0 2024-09-10T20:16:55.115 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:16:55.127 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-10T20:16:55.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-10T20:16:55.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.422291+0000 34'58 90:192 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:04:33.155670+0000 3 0 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.423630+0000 42'59 90:174 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:34:40.314710+0000 2 0 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:38.411995+0000 34'59 90:173 [2,1,0] 2 [2,1,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:13:50.365933+0000 4 0 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.426665+0000 41'58 90:189 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:07:08.856221+0000 3 0 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:16:38.411422+0000 32'57 90:161 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:37.778164+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:28:51.968866+0000 3 0 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 1 1 0 0 3085 0 0 17 42 17 active+recovery_wait+degraded 2024-09-10T20:16:38.413516+0000 40'59 90:186 [1,2,0] 1 [1,2,0] 1 47'60 2024-09-10T20:14:35.710973+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:55.564 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 active+clean 2024-09-10T20:16:38.425775+0000 41'60 90:177 [1,2,0] 1 [1,2,0] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:21:25.992715+0000 2 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:38.418282+0000 32'57 90:154 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:59:39.684025+0000 2 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 5 0 0 0 0 3085 0 0 19 42 19 active+clean 2024-09-10T20:16:38.418873+0000 85'61 90:176 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:57:21.285818+0000 4 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 active+clean 2024-09-10T20:16:38.412866+0000 40'60 90:176 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:50:33.196846+0000 5 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:16:38.424465+0000 34'58 90:139 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:04:07.120421+0000 2 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:16:38.424578+0000 42'59 90:160 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:45:16.208997+0000 5 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 active+clean 2024-09-10T20:16:38.411312+0000 35'58 90:180 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:39.306678+0000 1 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.420037+0000 39'58 90:148 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:28:37.983024+0000 2 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 active+clean 2024-09-10T20:16:38.423641+0000 38'59 90:169 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:57:52.444278+0000 0 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:38.428524+0000 32'57 90:143 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:44:24.326360+0000 2 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.424686+0000 41'58 90:153 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:11:16.865651+0000 2 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.421157+0000 40'59 90:179 [2,1,0] 2 [2,1,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:16:46.401070+0000 3 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 1 1 0 0 2818 0 0 18 42 18 active+recovery_wait+degraded 2024-09-10T20:16:38.421416+0000 78'60 90:175 [1,2,0] 1 [1,2,0] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 no scrub is scheduled 4 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:38.410721+0000 34'58 90:186 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:11:22.364507+0000 4 0 2024-09-10T20:16:55.565 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.418775+0000 41'58 90:185 [0,2,1] 0 [0,2,1] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T08:04:05.132236+0000 3 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1851 0 0 15 42 15 active+recovering+degraded 2024-09-10T20:16:38.430779+0000 32'57 90:176 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 3 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:38.415795+0000 32'57 90:176 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:50:34.936988+0000 1 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.412811+0000 41'59 90:163 [1,0,2] 1 [1,0,2] 1 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:01:59.892067+0000 3 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.425534+0000 41'59 90:182 [2,0,1] 2 [2,0,1] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:36:19.679805+0000 3 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.421340+0000 34'58 90:178 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:08:02.063750+0000 3 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:38.423832+0000 34'59 90:164 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:07:51.806964+0000 4 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:38.420352+0000 32'57 90:167 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:59:58.118652+0000 1 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.428065+0000 41'59 90:190 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:16:27.009391+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:20:36.093469+0000 3 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:38.426213+0000 32'57 90:152 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:16:10.017660+0000 2 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:38.414816+0000 38'58 90:175 [1,2,0] 1 [1,2,0] 1 38'58 2024-09-10T20:14:38.741886+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:21:22.465788+0000 4 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 3085 0 0 17 42 17 active+recovery_wait+degraded 2024-09-10T20:16:38.419528+0000 41'59 90:200 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:55.566 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:55.567 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:55.567 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:55.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-10T20:16:55.581 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-10T20:16:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:16:55.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-10T20:16:55.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-10T20:16:55.902 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:16:55.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-10T20:16:55.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-10T20:16:56.075 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:16:56.090 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-10T20:16:56.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-10T20:16:56.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-10T20:16:56.522 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:16:56.522 INFO:tasks.workunit.client.0.smithi039.stdout:1.1f 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.422291+0000 34'58 90:192 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:21.539395+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:04:33.155670+0000 3 0 2024-09-10T20:16:56.522 INFO:tasks.workunit.client.0.smithi039.stdout:1.1e 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.423630+0000 42'59 90:174 [1,2,0] 1 [1,2,0] 1 34'58 2024-09-10T20:14:15.483499+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:34:40.314710+0000 2 0 2024-09-10T20:16:56.522 INFO:tasks.workunit.client.0.smithi039.stdout:1.1d 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:38.411995+0000 34'59 90:173 [2,1,0] 2 [2,1,0] 2 34'59 2024-09-10T20:14:18.312343+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:13:50.365933+0000 4 0 2024-09-10T20:16:56.522 INFO:tasks.workunit.client.0.smithi039.stdout:1.b 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.426665+0000 41'58 90:189 [1,0,2] 1 [1,0,2] 1 41'58 2024-09-10T20:14:22.467324+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:07:08.856221+0000 3 0 2024-09-10T20:16:56.522 INFO:tasks.workunit.client.0.smithi039.stdout:1.a 3 0 0 0 0 1851 0 0 15 42 15 active+clean 2024-09-10T20:16:38.411422+0000 32'57 90:161 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:37.778164+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:28:51.968866+0000 3 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.9 5 1 1 0 0 3085 0 0 17 42 17 active+recovery_wait+degraded 2024-09-10T20:16:38.413516+0000 40'59 90:186 [1,2,0] 1 [1,2,0] 1 47'60 2024-09-10T20:14:35.710973+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.8 3 0 0 0 0 1851 0 0 18 42 18 active+clean 2024-09-10T20:16:38.425775+0000 41'60 90:177 [1,2,0] 1 [1,2,0] 1 36'59 2024-09-10T20:14:19.540697+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:21:25.992715+0000 2 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.7 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:38.418282+0000 32'57 90:154 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:23.498207+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:59:39.684025+0000 2 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.6 5 0 0 0 0 3085 0 0 19 42 19 active+clean 2024-09-10T20:16:38.418873+0000 85'61 90:176 [1,0,2] 1 [1,0,2] 1 36'60 2024-09-10T20:14:25.648101+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:57:21.285818+0000 4 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.5 6 0 0 0 0 3702 0 0 18 42 18 active+clean 2024-09-10T20:16:38.412866+0000 40'60 90:176 [2,0,1] 2 [2,0,1] 2 34'59 2024-09-10T20:14:20.339220+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:50:33.196846+0000 5 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.4 2 0 0 0 0 1234 0 0 16 42 16 active+clean 2024-09-10T20:16:38.424465+0000 34'58 90:139 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:16.430476+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T03:04:07.120421+0000 2 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.3 5 0 0 0 0 3085 0 0 17 42 17 active+clean 2024-09-10T20:16:38.424578+0000 42'59 90:160 [1,2,0] 1 [1,2,0] 1 42'59 2024-09-10T20:14:24.566037+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:45:16.208997+0000 5 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.2 1 0 0 0 0 617 0 0 16 42 16 active+clean 2024-09-10T20:16:38.411312+0000 35'58 90:180 [0,1,2] 0 [0,1,2] 0 35'58 2024-09-10T20:14:21.045563+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:54:39.306678+0000 1 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.c 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.420037+0000 39'58 90:148 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:17.392806+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T05:28:37.983024+0000 2 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.1 2 0 0 0 0 1234 0 0 17 42 17 active+clean 2024-09-10T20:16:38.423641+0000 38'59 90:169 [2,0,1] 2 [2,0,1] 2 32'57 2024-09-10T20:14:14.423908+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T21:57:52.444278+0000 0 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.1c 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:38.428524+0000 32'57 90:143 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:18.414276+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T07:44:24.326360+0000 2 0 2024-09-10T20:16:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.424686+0000 41'58 90:153 [1,0,2] 1 [1,0,2] 1 32'57 2024-09-10T20:14:12.531786+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T01:11:16.865651+0000 2 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.d 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.421157+0000 40'59 90:179 [2,1,0] 2 [2,1,0] 2 40'59 2024-09-10T20:14:21.296673+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:16:46.401070+0000 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.e 4 1 1 0 0 2818 0 0 18 42 18 active+recovery_wait+degraded 2024-09-10T20:16:38.421416+0000 78'60 90:175 [1,2,0] 1 [1,2,0] 1 39'59 2024-09-10T20:14:27.538028+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 no scrub is scheduled 4 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.f 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:38.410721+0000 34'58 90:186 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:28.517768+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:11:22.364507+0000 4 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.10 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.418775+0000 41'58 90:185 [0,2,1] 0 [0,2,1] 0 41'58 2024-09-10T20:14:29.559688+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-12T08:04:05.132236+0000 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.11 3 0 1 0 0 1851 0 0 15 42 15 active+recovering+degraded 2024-09-10T20:16:38.430779+0000 32'57 90:176 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:22.217071+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.12 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:38.415795+0000 32'57 90:176 [0,1,2] 0 [0,1,2] 0 32'57 2024-09-10T20:14:23.028209+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T20:50:34.936988+0000 1 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.13 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.412811+0000 41'59 90:163 [1,0,2] 1 [1,0,2] 1 41'59 2024-09-10T20:14:30.573887+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T04:01:59.892067+0000 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.14 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.425534+0000 41'59 90:182 [2,0,1] 2 [2,0,1] 2 41'59 2024-09-10T20:14:31.602223+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T23:36:19.679805+0000 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.15 3 0 0 0 0 1851 0 0 16 42 16 active+clean 2024-09-10T20:16:38.421340+0000 34'58 90:178 [2,0,1] 2 [2,0,1] 2 34'58 2024-09-10T20:14:23.241522+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T06:08:02.063750+0000 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.16 4 0 0 0 0 2468 0 0 17 42 17 active+clean 2024-09-10T20:16:38.423832+0000 34'59 90:164 [1,0,2] 1 [1,0,2] 1 34'59 2024-09-10T20:14:26.540160+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-11T21:07:51.806964+0000 4 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.17 1 0 0 0 0 617 0 0 15 42 15 active+clean 2024-09-10T20:16:38.420352+0000 32'57 90:167 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:33.683743+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:59:58.118652+0000 1 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.18 3 0 0 0 0 1851 0 0 17 42 17 active+clean 2024-09-10T20:16:38.428065+0000 41'59 90:190 [1,2,0] 1 [1,2,0] 1 41'59 2024-09-10T20:16:27.009391+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 periodic scrub scheduled @ 2024-09-12T00:20:36.093469+0000 3 0 2024-09-10T20:16:56.524 INFO:tasks.workunit.client.0.smithi039.stdout:1.19 2 0 0 0 0 1234 0 0 15 42 15 active+clean 2024-09-10T20:16:38.426213+0000 32'57 90:152 [1,2,0] 1 [1,2,0] 1 32'57 2024-09-10T20:14:13.451891+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:16:10.017660+0000 2 0 2024-09-10T20:16:56.525 INFO:tasks.workunit.client.0.smithi039.stdout:1.1a 4 0 0 0 0 2468 0 0 16 42 16 active+clean 2024-09-10T20:16:38.414816+0000 38'58 90:175 [1,2,0] 1 [1,2,0] 1 38'58 2024-09-10T20:14:38.741886+0000 0'0 2024-09-10T20:13:46.218219+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:21:22.465788+0000 4 0 2024-09-10T20:16:56.525 INFO:tasks.workunit.client.0.smithi039.stdout:1.1b 5 1 1 0 0 3085 0 0 17 42 17 active+recovery_wait+degraded 2024-09-10T20:16:38.419528+0000 41'59 90:200 [1,0,2] 1 [1,0,2] 1 34'58 2024-09-10T20:14:20.451747+0000 0'0 2024-09-10T20:13:46.218219+0000 0 1 no scrub is scheduled 4 0 2024-09-10T20:16:56.525 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:16:56.525 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:16:56.525 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:16:56.538 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-10T20:16:56.540 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-10T20:16:56.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:16:56.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-10T20:16:56.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:16:56.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-10T20:16:56.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:16:56.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-10T20:16:56.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:16:56.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-10T20:16:56.694 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-10T20:16:56.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:16:56.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-10T20:16:56.728 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-10T20:16:56.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-10T20:16:56.730 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-10T20:16:56.763 INFO:tasks.workunit.client.0.smithi039.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-10T20:16:56.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-10T20:16:56.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-10T20:16:56.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:16:56.786 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 47'60 (31'2) modify 1:97db2a0f:::existing_2:head by client.4629.0:1 2024-09-10T20:14:33.967866+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:16:56.786 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 20 _merge_object_divergent_entries: keeping 47'60 (31'2) modify 1:97db2a0f:::existing_2:head by client.4629.0:1 2024-09-10T20:14:33.967866+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 47'60 last_divergent_update: 47'60 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 47'60 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 47'60 (31'2) modify 1:97db2a0f:::existing_2:head by client.4629.0:1 2024-09-10T20:14:33.967866+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'42 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 47'61 (31'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-10T20:14:33.987166+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 20 _merge_object_divergent_entries: keeping 47'61 (31'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-10T20:14:33.987166+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:16:56.787 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 47'61 last_divergent_update: 47'61 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 47'61 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 47'61 (31'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-10T20:14:33.987166+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'42 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 47'60 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-09-10T20:14:33.956614+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:16:56.788 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 20 _merge_object_divergent_entries: keeping 47'60 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-09-10T20:14:33.956614+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 47'60 last_divergent_update: 47'60 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.075+0000 7f3d3db3c640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 47'61 (31'4) delete 1:70b3bf12:::existing_4:head by client.4640.0:1 2024-09-10T20:14:33.991377+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 20 _merge_object_divergent_entries: keeping 47'61 (31'4) delete 1:70b3bf12:::existing_4:head by client.4640.0:1 2024-09-10T20:14:33.991377+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 47'61 last_divergent_update: 47'61 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: more recent entry found: 78'60 (31'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-09-10T20:15:39.096420+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 47'60 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-09-10T20:14:33.981103+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:16:56.789 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 20 _merge_object_divergent_entries: keeping 47'60 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-09-10T20:14:33.981103+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:16:56.796 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 47'60 last_divergent_update: 47'60 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.079+0000 7f3d3bb38640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 47'58 (31'3) delete 1:8881e6d8:::existing_3:head by client.4641.0:1 2024-09-10T20:14:33.990457+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 20 _merge_object_divergent_entries: keeping 47'58 (31'3) delete 1:8881e6d8:::existing_3:head by client.4641.0:1 2024-09-10T20:14:33.990457+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 47'58 last_divergent_update: 47'58 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 47'58 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 47'58 (31'3) delete 1:8881e6d8:::existing_3:head by client.4641.0:1 2024-09-10T20:14:33.990457+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-10T20:16:56.797 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:16:26.299+0000 7f3d3cb3a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'42 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 47'58 (31'3) delete 1:8881e6d8:::existing_3:head by client.4641.0:1 2024-09-10T20:14:33.990457+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 20 _merge_object_divergent_entries: keeping 47'58 (31'3) delete 1:8881e6d8:::existing_3:head by client.4641.0:1 2024-09-10T20:14:33.990457+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 47'58 last_divergent_update: 47'58 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 47'58 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 47'58 (31'3) delete 1:8881e6d8:::existing_3:head by client.4641.0:1 2024-09-10T20:14:33.990457+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-10T20:16:56.798 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.0.log:2024-09-10T20:16:26.079+0000 7f0cb56a6640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'42 2024-09-10T20:16:56.799 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-10T20:16:56.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:16:56.819 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-10T20:16:56.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:16:56.838 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-10T20:16:56.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-10T20:16:56.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.27146.out 2024-09-10T20:16:56.839 INFO:tasks.workunit.client.0.smithi039.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27146.out': No such file or directory 2024-09-10T20:16:56.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-10T20:16:56.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-10T20:16:56.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-10T20:16:57.369 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-10T20:16:57.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-10T20:16:57.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:16:57.386 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:16:57.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:16:57.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:16:57.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:17:02.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:17:02.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-10T20:17:02.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:17:02.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:17:02.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:17:02.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:17:02.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:17:02.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:17:02.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:17:02.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:17:02.715 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:17:02.715 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:17:02.716 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:17:02.716 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:17:02.717 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:17:02.718 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:17:02.718 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:17:02.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:17:02.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:17:02.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:17:02.719 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:17:02.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:17:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:17:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:17:02.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:17:02.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:02.760 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:02.760 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/divergent-priors 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/divergent-priors 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:17:02.762 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:17:02.763 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:17:02.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:17:02.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:17:02.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:17:02.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:17:02.764 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:17:02.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:17:02.765 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:17:02.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:17:02.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:17:02.766 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:17:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:17:02.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:17:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:17:02.768 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:17:02.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:17:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:17:02.769 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:17:02.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:17:02.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:02.771 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:17:02.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:17:02.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:17:02.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/divergent-priors 2024-09-10T20:17:02.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:17:02.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:02.773 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:02.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27146 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-10T20:17:02.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-10T20:17:02.776 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-10T20:17:02.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-10T20:17:02.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-10T20:17:02.777 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:17:02.777 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:17:02.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-10T20:17:02.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/divergent-priors 2024-09-10T20:17:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:17:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:17:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:17:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/divergent-priors/a 2024-09-10T20:17:02.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-10T20:17:03.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:17:03.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:17:03.258 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:17:03.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:17:03.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:03.258 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:03.259 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:17:03.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:17:03.469 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:17:03.470 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:17:03.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:17:03.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:17:03.471 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:03.473 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:03.474 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:17:03.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:17:03.474 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get fsid 2024-09-10T20:17:03.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:17:03.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:17:03.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:17:03.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:17:03.592 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:17:03.592 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:17:03.592 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:17:03.592 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:17:03.592 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:17:03.592 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:03.593 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:03.593 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-mon.a.asok 2024-09-10T20:17:03.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:17:03.593 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27146/ceph-mon.a.asok config get mon_host 2024-09-10T20:17:03.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-10T20:17:03.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/divergent-priors 2024-09-10T20:17:03.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:17:03.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:17:03.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:17:03.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/divergent-priors/x 2024-09-10T20:17:03.698 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:17:04.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:17:04.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:17:04.609 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:17:04.610 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:17:04.610 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:04.610 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:04.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:17:04.610 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:17:04.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:17:04.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-10T20:17:04.674 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:17:04.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:04.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:17:04.677 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:17:04.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:17:04.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:17:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:17:04.679 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 af440182-d88c-48fb-b1e2-3030d54ba58c' 2024-09-10T20:17:04.679 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:17:04.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:17:04.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAqeBmNx/5KRAAPjSm5QN3JZKSzC+jJfcksg== 2024-09-10T20:17:04.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAqeBmNx/5KRAAPjSm5QN3JZKSzC+jJfcksg=="}' 2024-09-10T20:17:04.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new af440182-d88c-48fb-b1e2-3030d54ba58c -i td/divergent-priors/0/new.json 2024-09-10T20:17:05.005 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:17:05.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/0/new.json 2024-09-10T20:17:05.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAqeBmNx/5KRAAPjSm5QN3JZKSzC+jJfcksg== --osd-uuid af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:17:05.066 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:05.066+0000 7feb2d0fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:05.073 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:05.074+0000 7feb2d0fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:05.074 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:05.074+0000 7feb2d0fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:05.075 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:05.074+0000 7feb2d0fa7c0 -1 bdev(0x55a75526dc00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:17:05.075 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:05.074+0000 7feb2d0fa7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-10T20:17:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-10T20:17:11.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:17:11.386 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:17:11.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:17:11.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:17:12.026 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:17:12.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:17:12.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:17:12.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:17:12.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:17:12.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:17:12.069 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:12.066+0000 7f317a95d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:12.071 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:12.070+0000 7f317a95d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:12.072 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:12.070+0000 7f317a95d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:17:12.519 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:17:12.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:12.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:13.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:13.071 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:13.070+0000 7f317a95d7c0 -1 Falling back to public interface 2024-09-10T20:17:14.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:14.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:14.020 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:17:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:17:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:14.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:14.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:14.734 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:14.734+0000 7f317a95d7c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:17:15.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:15.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:15.470 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:17:15.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:17:15.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:15.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:15.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:16.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:16.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:16.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:17:16.963 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:17:16.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:16.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:17.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:18.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:18.396 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:17:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:17:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:18.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:18.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:19.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:19.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:19.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:17:19.894 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:17:19.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:19.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:20.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:21.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:21.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:21.381 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:17:21.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:17:21.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:21.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:21.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:22.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:22.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:22.893 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:17:22.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:17:22.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:22.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:23.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:24.356 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:17:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:17:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:24.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:25.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:25.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:25.947 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:17:25.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:17:25.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:25.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:26.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:27.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:27.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:27.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:17:27.422 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:17:27.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:27.422 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:28.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:29.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:29.396 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:29.397 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:17:29.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:17:29.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:29.397 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:30.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:31.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:31.211 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:17:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:17:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:31.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:32.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:33.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:33.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:33.139 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:17:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:17:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:33.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:34.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:35.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:35.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:35.173 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:17:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:17:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:35.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:36.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:37.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:37.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:37.041 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:17:37.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:17:37.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:37.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:37.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:38.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:38.688 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:17:38.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:17:38.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:38.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:17:39.153 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/907649505,v1:127.0.0.1:6803/907649505] [v2:127.0.0.1:6804/907649505,v1:127.0.0.1:6805/907649505] exists,up af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:17:39.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:17:39.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:17:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:17:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-10T20:17:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-10T20:17:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:17:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:17:39.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:17:39.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:17:39.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:17:39.163 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:17:39.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6e593c3e-6dfc-43a0-8d93-a3049d9fe32b 2024-09-10T20:17:39.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b' 2024-09-10T20:17:39.164 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b 2024-09-10T20:17:39.165 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:17:39.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBjqeBmqDt2CxAAkZvqsK88Fy786GrIsU5aYQ== 2024-09-10T20:17:39.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBjqeBmqDt2CxAAkZvqsK88Fy786GrIsU5aYQ=="}' 2024-09-10T20:17:39.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b -i td/divergent-priors/1/new.json 2024-09-10T20:17:39.798 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:17:39.816 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/1/new.json 2024-09-10T20:17:39.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjqeBmqDt2CxAAkZvqsK88Fy786GrIsU5aYQ== --osd-uuid 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b 2024-09-10T20:17:39.854 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:39.854+0000 7f79444997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:39.856 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:39.854+0000 7f79444997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:39.857 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:39.858+0000 7f79444997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:39.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:39.858+0000 7f79444997c0 -1 bdev(0x5588eb5e9c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:17:39.858 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:39.858+0000 7f79444997c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-10T20:17:45.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-10T20:17:45.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:17:45.505 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:17:45.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:17:45.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:17:46.115 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:17:46.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:17:46.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:17:46.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:17:46.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:17:46.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:17:46.157 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:46.158+0000 7f1c58c247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:46.159 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:46.158+0000 7f1c58c247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:46.161 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:46.162+0000 7f1c58c247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:17:46.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:17:46.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:17:46.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:17:46.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:17:46.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:17:46.595 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:46.596 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:17:46.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:17:46.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:46.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:47.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:47.416 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:47.414+0000 7f1c58c247c0 -1 Falling back to public interface 2024-09-10T20:17:48.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:48.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:48.107 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:17:48.107 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:17:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:48.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:48.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:49.033 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:17:49.030+0000 7f1c58c247c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:17:49.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:49.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:49.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:17:49.577 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:17:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:49.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:50.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:51.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:51.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:51.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:17:51.272 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:17:51.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:51.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:51.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:52.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:52.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:52.934 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:17:52.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:17:52.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:52.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:53.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:54.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:54.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:54.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:17:54.403 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:17:54.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:54.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:54.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:55.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:55.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:55.924 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:17:55.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:17:55.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:55.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:56.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:57.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:57.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:57.449 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:17:57.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:17:57.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:57.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:58.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:17:59.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:17:59.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:17:59.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:17:59.141 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:17:59.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:17:59.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:17:59.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:00.596 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:00.596 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:18:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:18:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:00.597 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:01.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:02.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:02.151 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:02.152 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:18:02.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:18:02.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:02.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:02.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:03.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:03.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:03.668 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:18:03.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:18:03.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:03.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:04.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:05.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:05.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:05.234 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:18:05.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:18:05.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:05.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:05.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:06.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:06.932 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:18:06.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:06.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:18:06.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:06.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:07.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:08.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:08.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:08.775 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:18:08.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:18:08.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:08.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:09.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:10.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:10.565 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:18:10.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:10.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:18:10.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:10.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:11.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:12.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:12.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:12.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:18:12.547 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:18:12.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:12.548 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:18:13.019 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1972346789,v1:127.0.0.1:6811/1972346789] [v2:127.0.0.1:6812/1972346789,v1:127.0.0.1:6813/1972346789] exists,up 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/divergent-priors 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:18:13.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:18:13.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:18:13.025 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:18:13.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:18:13.027 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:18:13.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 8d54aebd-dd27-4eb6-bbb3-ec628062ac61' 2024-09-10T20:18:13.027 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:18:13.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFqeBmYo0NAxAA5a0snRIPmHpgIyC0zxYEcg== 2024-09-10T20:18:13.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFqeBmYo0NAxAA5a0snRIPmHpgIyC0zxYEcg=="}' 2024-09-10T20:18:13.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 -i td/divergent-priors/2/new.json 2024-09-10T20:18:13.647 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:18:13.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/divergent-priors/2/new.json 2024-09-10T20:18:13.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFqeBmYo0NAxAA5a0snRIPmHpgIyC0zxYEcg== --osd-uuid 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:18:13.702 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:13.701+0000 7f50b8f9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:18:13.728 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:13.725+0000 7f50b8f9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:18:13.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:13.729+0000 7f50b8f9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:18:13.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:13.729+0000 7f50b8f9d7c0 -1 bdev(0x5595db163c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:18:13.729 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:13.729+0000 7f50b8f9d7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-10T20:18:19.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-10T20:18:19.553 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:18:19.554 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:18:19.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:18:19.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:18:20.182 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:18:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:18:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:18:20.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:18:20.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:18:20.184 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:18:20.218 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:20.217+0000 7f556266d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:18:20.220 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:20.217+0000 7f556266d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:18:20.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:20.221+0000 7f556266d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:18:20.658 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:18:20.659 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:18:20.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:20.660 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:21.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:21.221 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:21.217+0000 7f556266d7c0 -1 Falling back to public interface 2024-09-10T20:18:22.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:22.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:22.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:18:22.150 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:18:22.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:22.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:22.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:23.128 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:18:23.125+0000 7f556266d7c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:18:23.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:23.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:23.614 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:18:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:18:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:23.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:24.140 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:25.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:25.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:25.142 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:18:25.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:18:25.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:25.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:25.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:26.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:26.601 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:26.602 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:18:26.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:18:26.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:26.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:27.059 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:28.060 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:28.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:28.061 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:18:28.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:18:28.061 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:28.062 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:28.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:29.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:29.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:29.639 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:18:29.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:18:29.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:29.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:30.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:31.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:31.408 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:31.409 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:18:31.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:18:31.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:31.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:31.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:32.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:32.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:32.973 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:18:32.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:18:32.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:32.974 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:33.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:34.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:34.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:34.449 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:18:34.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:18:34.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:34.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:35.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:36.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:36.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:36.336 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:18:36.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:18:36.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:36.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:37.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:38.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:38.565 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:18:38.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:18:38.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:38.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:39.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:40.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:40.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:40.528 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:18:40.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:18:40.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:40.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:41.530 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:42.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:42.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:42.533 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:18:42.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:18:42.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:42.533 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:43.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:44.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:44.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:44.527 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:18:44.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:18:44.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:44.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:45.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:18:46.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:18:46.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:18:46.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:18:46.411 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:18:46.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:18:46.412 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:18:46.899 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2787399546,v1:127.0.0.1:6819/2787399546] [v2:127.0.0.1:6820/2787399546,v1:127.0.0.1:6821/2787399546] exists,up 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:18:46.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:18:46.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:18:46.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:18:46.900 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-10T20:18:47.564 INFO:tasks.workunit.client.0.smithi039.stderr:noout is set 2024-09-10T20:18:47.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-10T20:18:48.376 INFO:tasks.workunit.client.0.smithi039.stderr:noin is set 2024-09-10T20:18:48.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-10T20:18:49.239 INFO:tasks.workunit.client.0.smithi039.stderr:nodown is set 2024-09-10T20:18:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-10T20:18:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: create_ec_pool: local pool_name=test 2024-09-10T20:18:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2444: create_ec_pool: shift 2024-09-10T20:18:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: create_ec_pool: local allow_overwrites=true 2024-09-10T20:18:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: create_ec_pool: shift 2024-09-10T20:18:49.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-10T20:18:49.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2450: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-10T20:18:49.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-10T20:18:50.673 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' already exists 2024-09-10T20:18:50.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:18:51.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: '[' true = true ']' 2024-09-10T20:18:51.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-10T20:18:52.445 INFO:tasks.workunit.client.0.smithi039.stderr:set pool 1 allow_ec_overwrites to true 2024-09-10T20:18:52.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: wait_for_clean 2024-09-10T20:18:52.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:18:52.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:18:52.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:18:52.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:18:52.466 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:18:52.466 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:18:52.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:18:52.466 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:18:52.467 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:18:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:18:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:18:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:18:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:18:52.588 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:18:52.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:18:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:18:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:18:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:18:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:18:53.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:18:53.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:18:53.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836502 2024-09-10T20:18:53.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836502 2024-09-10T20:18:53.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502' 2024-09-10T20:18:53.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:18:53.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:18:53.398 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-10T20:18:53.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-10T20:18:53.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-09-10T20:18:53.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:18:53.399 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:18:53.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-10T20:18:53.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-10T20:18:53.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509448' 2024-09-10T20:18:53.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:18:53.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836502 2024-09-10T20:18:53.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:18:53.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:18:53.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836502 2024-09-10T20:18:53.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:18:53.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836502 2024-09-10T20:18:53.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-10T20:18:53.571 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836502 2024-09-10T20:18:53.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:18:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836502 2024-09-10T20:18:54.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:18:54.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-10T20:18:54.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:18:54.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:18:54.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-10T20:18:54.018 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:18:54.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-10T20:18:54.019 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672976 2024-09-10T20:18:54.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-10T20:18:54.020 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:18:54.504 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-10T20:18:54.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:18:54.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-10T20:18:54.505 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:18:54.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:18:54.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-10T20:18:54.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:18:54.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-10T20:18:54.509 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509448 2024-09-10T20:18:54.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-10T20:18:54.510 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:18:54.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-10T20:18:54.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:18:54.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:18:54.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:18:55.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:18:55.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:18:55.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:18:55.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:18:55.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:18:55.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:18:55.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:18:55.604 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:18:56.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:18:56.049 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:18:56.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:18:56.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:18:56.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:18:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:18:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:18:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: return 0 2024-09-10T20:18:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-10T20:18:56.643 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:18:56.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:18:57.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:18:57.129 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:18:57.129 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:18:57.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:18:57.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:18:57.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:18:57.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836505 2024-09-10T20:18:57.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836505 2024-09-10T20:18:57.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505' 2024-09-10T20:18:57.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:18:57.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:18:57.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672979 2024-09-10T20:18:57.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672979 2024-09-10T20:18:57.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672979' 2024-09-10T20:18:57.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:18:57.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:18:57.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509451 2024-09-10T20:18:57.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509451 2024-09-10T20:18:57.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836505 1-42949672979 2-64424509451' 2024-09-10T20:18:57.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:18:57.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:18:57.646 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836505 2024-09-10T20:18:57.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:18:57.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836505 2024-09-10T20:18:57.647 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:18:57.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836505 2024-09-10T20:18:57.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-10T20:18:57.649 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836505 2024-09-10T20:18:57.649 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:18:58.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-10T20:18:58.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:18:58.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672979 2024-09-10T20:18:58.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:18:58.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:18:58.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672979 2024-09-10T20:18:58.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:18:58.154 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672979 2024-09-10T20:18:58.154 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672979 2024-09-10T20:18:58.155 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-10T20:18:58.155 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:18:58.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-10T20:18:58.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:18:58.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509451 2024-09-10T20:18:58.642 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:18:58.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:18:58.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509451 2024-09-10T20:18:58.643 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:18:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509451 2024-09-10T20:18:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-10T20:18:58.645 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509451 2024-09-10T20:18:58.645 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:18:59.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509450 -lt 64424509451 2024-09-10T20:18:59.137 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:19:00.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:19:00.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:19:00.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-10T20:19:00.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-10T20:19:00.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:19:00.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:19:00.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:19:00.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:19:00.639 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:19:00.639 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:19:00.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:19:00.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:19:00.640 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:19:00.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:19:00.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:19:00.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:19:00.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:19:00.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:19:00.756 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:19:01.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:19:01.248 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:19:01.248 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:19:01.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:19:01.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:19:01.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:19:01.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836508 2024-09-10T20:19:01.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836508 2024-09-10T20:19:01.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508' 2024-09-10T20:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:19:01.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:19:01.598 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-10T20:19:01.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-10T20:19:01.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-09-10T20:19:01.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:19:01.599 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:19:01.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-10T20:19:01.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-10T20:19:01.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509454' 2024-09-10T20:19:01.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:19:01.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836508 2024-09-10T20:19:01.767 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:19:01.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:19:01.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836508 2024-09-10T20:19:01.769 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:19:01.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836508 2024-09-10T20:19:01.770 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836508 2024-09-10T20:19:01.770 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-10T20:19:01.770 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:19:02.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-10T20:19:02.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:19:02.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-10T20:19:02.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:19:02.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:19:02.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-10T20:19:02.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:19:02.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-10T20:19:02.250 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-10T20:19:02.251 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672982 2024-09-10T20:19:02.251 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:19:02.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-10T20:19:02.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:19:02.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-10T20:19:02.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:19:02.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:19:02.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-10T20:19:02.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:19:02.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-10T20:19:02.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-10T20:19:02.727 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509454 2024-09-10T20:19:02.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:19:03.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509452 -lt 64424509454 2024-09-10T20:19:03.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:19:04.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:19:04.220 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:19:04.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-10T20:19:04.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:19:04.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:19:04.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:19:05.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:19:05.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:19:05.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:19:05.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:19:05.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:19:05.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:19:05.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:19:05.314 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:19:05.734 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:19:05.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:19:05.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:19:05.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:19:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:19:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:19:06.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:19:06.321 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-10T20:19:06.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-10T20:19:06.739 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:19:06.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-10T20:19:06.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-10T20:19:06.757 INFO:tasks.workunit.client.0.smithi039.stdout:primary and soon to be divergent is 1 2024-09-10T20:19:06.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:19:07.172 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:19:07.173 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T20:18:51.562239+0000 0'0 29:16 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:24:56.565633+0000 0 0 2024-09-10T20:19:07.173 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:19:07.173 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:19:07.173 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:19:07.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-10T20:19:07.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-10T20:19:07.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-10T20:19:07.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-10T20:19:07.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-10T20:19:07.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-10T20:19:07.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-10T20:19:07.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-10T20:19:07.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-10T20:19:07.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-10T20:19:07.190 INFO:tasks.workunit.client.0.smithi039.stdout:writing initial objects 2024-09-10T20:19:07.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-10T20:19:07.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-10T20:19:07.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:07.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-10T20:19:07.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:07.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-10T20:19:07.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:07.609 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-10T20:19:07.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:07.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-10T20:19:08.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:08.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-10T20:19:08.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:08.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-10T20:19:08.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:08.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-10T20:19:08.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:08.666 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-10T20:19:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-10T20:19:09.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:09.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-10T20:19:09.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:09.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-10T20:19:09.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:09.513 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-10T20:19:09.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:09.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-10T20:19:09.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:09.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-10T20:19:10.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:10.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-10T20:19:10.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:10.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-10T20:19:10.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:10.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-10T20:19:10.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:10.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-10T20:19:11.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:11.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-10T20:19:11.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:11.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-10T20:19:11.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:11.418 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-10T20:19:11.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:11.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-10T20:19:11.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:11.845 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-10T20:19:12.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:12.099 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-10T20:19:12.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:12.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-10T20:19:12.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:12.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-10T20:19:12.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:12.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-10T20:19:13.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:13.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-10T20:19:13.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:13.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-10T20:19:13.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:13.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-10T20:19:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:13.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-10T20:19:13.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:13.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-10T20:19:14.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:14.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-10T20:19:14.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:14.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-10T20:19:14.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:14.503 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-10T20:19:14.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:14.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-10T20:19:14.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:14.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-10T20:19:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:15.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-10T20:19:15.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:15.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-10T20:19:15.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:15.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-10T20:19:15.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:15.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-10T20:19:15.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:15.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-10T20:19:16.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:16.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-10T20:19:16.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:16.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-10T20:19:16.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:16.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-10T20:19:16.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:16.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-10T20:19:17.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:17.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-10T20:19:17.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:17.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-10T20:19:17.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:17.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-10T20:19:17.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:17.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-10T20:19:18.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:18.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-10T20:19:18.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:18.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-10T20:19:18.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:18.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-10T20:19:18.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:18.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-10T20:19:18.963 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:18.964 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-10T20:19:19.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:19.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-10T20:19:19.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:19.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-10T20:19:19.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:19.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-10T20:19:19.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:19.778 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-10T20:19:19.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:19.988 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-10T20:19:20.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:20.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-10T20:19:20.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:20.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-10T20:19:20.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:20.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-10T20:19:20.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:20.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-10T20:19:21.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:21.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-10T20:19:21.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:21.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-10T20:19:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:21.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-10T20:19:21.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:21.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-10T20:19:21.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:21.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-10T20:19:22.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:22.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-10T20:19:22.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:22.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-10T20:19:22.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:22.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-10T20:19:22.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:22.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-10T20:19:23.056 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:23.057 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-10T20:19:23.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:23.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-10T20:19:23.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:23.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-10T20:19:23.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:23.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-10T20:19:23.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:23.872 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-10T20:19:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:24.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-10T20:19:24.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:24.348 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-10T20:19:24.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:24.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-10T20:19:24.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:24.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-10T20:19:24.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:24.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-10T20:19:25.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:25.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-10T20:19:25.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:25.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-10T20:19:25.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:25.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-10T20:19:25.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:25.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-10T20:19:26.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:26.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-10T20:19:26.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:26.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-10T20:19:26.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:26.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-10T20:19:26.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:26.764 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-10T20:19:26.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:26.947 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-10T20:19:27.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:27.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-10T20:19:27.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:27.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-10T20:19:27.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:27.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-10T20:19:27.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:27.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-10T20:19:28.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:28.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-10T20:19:28.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:28.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-10T20:19:28.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:28.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-10T20:19:28.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-10T20:19:28.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-10T20:19:28.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:19:28.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-10T20:19:28.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:19:28.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:19:28.877 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:19:28.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:19:28.877 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:19:28.877 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:19:28.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:19:28.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:19:28.878 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:19:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:19:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:19:28.955 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:19:28.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:19:28.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:19:28.956 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:19:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:19:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:19:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:19:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:19:29.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:19:29.472 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:19:29.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836516 2024-09-10T20:19:29.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836516 2024-09-10T20:19:29.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516' 2024-09-10T20:19:29.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:19:29.650 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:19:29.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672990 2024-09-10T20:19:29.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672990 2024-09-10T20:19:29.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-42949672990' 2024-09-10T20:19:29.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:19:29.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:19:29.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509461 2024-09-10T20:19:29.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509461 2024-09-10T20:19:29.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836516 1-42949672990 2-64424509461' 2024-09-10T20:19:29.998 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:19:29.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:19:29.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836516 2024-09-10T20:19:30.000 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:19:30.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836516 2024-09-10T20:19:30.000 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:19:30.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836516 2024-09-10T20:19:30.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-09-10T20:19:30.003 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 21474836516 2024-09-10T20:19:30.003 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:19:30.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836515 -lt 21474836516 2024-09-10T20:19:30.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:19:31.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:19:31.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:19:31.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836516 -lt 21474836516 2024-09-10T20:19:31.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:19:31.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:19:31.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672990 2024-09-10T20:19:31.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:19:31.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672990 2024-09-10T20:19:31.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:19:31.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672990 2024-09-10T20:19:31.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-09-10T20:19:31.979 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 42949672990 2024-09-10T20:19:31.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:19:32.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672990 -lt 42949672990 2024-09-10T20:19:32.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:19:32.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509461 2024-09-10T20:19:32.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:19:32.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:19:32.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509461 2024-09-10T20:19:32.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:19:32.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509461 2024-09-10T20:19:32.452 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 64424509461 2024-09-10T20:19:32.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509461' 2024-09-10T20:19:32.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:19:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509462 -lt 64424509461 2024-09-10T20:19:32.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:19:32.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:19:32.977 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:19:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:19:33.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:19:33.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:19:33.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:19:33.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:19:33.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:19:33.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:19:33.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:19:34.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:19:34.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:19:34.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:19:34.002 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:19:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:19:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:19:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:19:34.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-10T20:19:34.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-10T20:19:34.575 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=existing_1 2024-09-10T20:19:34.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test existing_1 2024-09-10T20:19:34.576 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:19:35.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-10T20:19:35.019 INFO:tasks.workunit.client.0.smithi039.stdout:blackholing osds 0 2 2024-09-10T20:19:35.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-10T20:19:35.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:19:35.430 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:19:35.430 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-09-10T20:18:51.562239+0000 29'100 29:116 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:24:56.565633+0000 0 0 2024-09-10T20:19:35.430 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:19:35.430 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:19:35.430 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:19:35.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:35.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-10T20:19:35.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:19:35.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:19:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:19:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:19:35.447 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:19:35.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:19:35.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-10T20:19:35.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-10T20:19:35.545 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:19:35.545 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:19:35.546 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:19:35.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:35.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-10T20:19:35.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:19:35.558 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:19:35.558 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:19:35.558 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:19:35.559 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:19:35.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:19:35.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-10T20:19:35.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-10T20:19:35.670 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:19:35.670 INFO:tasks.workunit.client.0.smithi039.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-10T20:19:35.670 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:19:35.684 INFO:tasks.workunit.client.0.smithi039.stdout:writing divergent object 2024-09-10T20:19:35.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-10T20:19:35.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-10T20:19:35.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-10T20:19:36.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-10T20:19:36.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-10T20:19:36.915 INFO:tasks.workunit.client.0.smithi039.stdout:created pool test snap snap1 2024-09-10T20:19:36.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-10T20:19:36.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-10T20:19:36.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-10T20:19:46.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-10T20:19:46.927 INFO:tasks.workunit.client.0.smithi039.stderr:rados: no process found 2024-09-10T20:19:46.928 INFO:tasks.workunit.client.0.smithi039.stdout:killing all the osds 2024-09-10T20:19:46.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-10T20:19:46.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:19:47.370 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:19:47.370 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-09-10T20:18:51.562239+0000 30'104 30:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:00:37.806214+0000 0 0 2024-09-10T20:19:47.370 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:19:47.370 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:19:47.370 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:19:47.387 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-10T20:19:47.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:19:47.389 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:19:47.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:19:47.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:19:47.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:19:47.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:19:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-10T20:19:47.498 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-10T20:19:48.162 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:19:48.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-10T20:19:48.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-10T20:19:48.755 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:19:48.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-10T20:19:48.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-10T20:19:49.333 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:19:49.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:49.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-10T20:19:49.919 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already out. 2024-09-10T20:19:49.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:49.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-10T20:19:50.484 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already out. 2024-09-10T20:19:50.502 INFO:tasks.workunit.client.0.smithi039.stdout:bringing up non_divergent 0 2 2024-09-10T20:19:50.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-10T20:19:50.502 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:19:50.938 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:19:50.938 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-09-10T20:18:51.562239+0000 30'104 30:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:00:37.806214+0000 0 0 2024-09-10T20:19:50.938 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:19:50.938 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:19:50.938 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:19:50.956 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:19:50.957 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:19:50.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:19:50.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:19:50.958 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:19:50.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:19:50.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:19:50.958 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:19:50.959 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:19:50.960 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:19:50.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:19:50.961 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:19:50.961 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:19:50.962 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:19:51.008 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:51.004+0000 7f8f5e0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:51.010 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:51.008+0000 7f8f5e0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:51.011 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:51.008+0000 7f8f5e0f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:51.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:19:51.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:19:51.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:19:51.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:19:51.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:19:51.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:19:51.420 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:19:51.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:19:51.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:19:51.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:19:51.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:19:52.255 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:52.252+0000 7f8f5e0f07c0 -1 Falling back to public interface 2024-09-10T20:19:52.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:19:52.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:19:52.892 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:19:52.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:19:52.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:19:52.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:19:53.349 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:19:54.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:19:54.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:19:54.351 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:19:54.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:19:54.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:19:54.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:19:54.407 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:54.404+0000 7f8f5e0f07c0 -1 osd.0 29 log_to_monitors true 2024-09-10T20:19:54.824 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/3163993053,v1:127.0.0.1:6803/3163993053] [v2:127.0.0.1:6804/3163993053,v1:127.0.0.1:6805/3163993053] exists,up af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:19:54.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:19:54.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:19:54.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:19:54.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:19:54.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:19:54.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:19:54.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:19:54.830 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:19:54.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:19:54.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:19:54.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-10T20:19:54.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:19:54.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:19:54.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:19:54.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:19:54.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:54.876+0000 7ff847bbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:54.902 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:54.900+0000 7ff847bbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:54.905 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:54.904+0000 7ff847bbb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:19:55.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:19:55.356 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:19:55.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:19:55.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:19:55.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:19:56.175 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:56.172+0000 7ff847bbb7c0 -1 Falling back to public interface 2024-09-10T20:19:56.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:19:56.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:19:56.884 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:19:56.884 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:19:56.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:19:56.885 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:19:57.350 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:19:58.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:19:58.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:19:58.353 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:19:58.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:19:58.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:19:58.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:19:58.373 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:19:58.372+0000 7ff847bbb7c0 -1 osd.2 29 log_to_monitors true 2024-09-10T20:19:59.000 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/720469603,v1:127.0.0.1:6811/720469603] [v2:127.0.0.1:6812/720469603,v1:127.0.0.1:6813/720469603] exists,up 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:19:59.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:19:59.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:19:59.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:19:59.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:59.001 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-10T20:19:59.591 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already in. 2024-09-10T20:19:59.607 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:19:59.608 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-10T20:20:00.188 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already in. 2024-09-10T20:20:00.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-10T20:20:05.207 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-10T20:20:05.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-10T20:20:05.209 INFO:tasks.workunit.client.0.smithi039.stdout:writing non-divergent object existing_4 2024-09-10T20:20:05.209 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-10T20:20:05.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:20:05.649 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:20:05.649 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-09-10T20:20:00.931290+0000 29'100 43:127 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:58:24.583328+0000 0 0 2024-09-10T20:20:05.649 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:20:05.649 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:20:05.649 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:20:05.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-10T20:20:05.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:20:05.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-10T20:20:05.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:20:05.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:20:05.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:20:05.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:20:05.828 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:20:05.828 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:20:05.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:20:05.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:20:05.828 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:20:05.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:20:05.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:20:05.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:20:05.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:20:05.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:20:05.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:20:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:20:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:20:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:20:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:20:06.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:20:06.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:20:06.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789957 2024-09-10T20:20:06.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789957 2024-09-10T20:20:06.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957' 2024-09-10T20:20:06.579 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:20:06.579 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:20:06.704 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:20:06.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:20:06.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:20:06.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:20:06.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:20:06.707 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:20:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=167503724548 2024-09-10T20:20:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 167503724548 2024-09-10T20:20:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-09-10T20:20:06.874 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:20:06.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-158913789957 2024-09-10T20:20:06.875 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:20:06.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:20:06.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-158913789957 2024-09-10T20:20:06.877 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:20:06.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789957 2024-09-10T20:20:06.879 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 158913789957 2024-09-10T20:20:06.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-09-10T20:20:06.879 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:20:07.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789957 2024-09-10T20:20:07.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:20:08.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:20:08.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:20:08.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789957 -lt 158913789957 2024-09-10T20:20:08.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:20:08.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-167503724548 2024-09-10T20:20:08.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:20:08.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:20:08.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-167503724548 2024-09-10T20:20:08.870 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:20:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=167503724548 2024-09-10T20:20:08.871 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 167503724548 2024-09-10T20:20:08.871 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-09-10T20:20:08.871 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:20:09.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 167503724548 -lt 167503724548 2024-09-10T20:20:09.482 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:20:09.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:09.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:20:10.105 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:10.106 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:10.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:10.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:10.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:10.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:10.107 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:10.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:10.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:10.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:10.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:20:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:20:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:20:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:11.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:20:11.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:11.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:11.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:11.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:11.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:11.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:11.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:11.249 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:11.689 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:11.690 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:11.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:11.691 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:12.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:12.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:12.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:12.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:12.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:12.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:12.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:12.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:12.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:12.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:20:12.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:12.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:20:13.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:13.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:13.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:13.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:13.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:13.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:13.094 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:13.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:13.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:13.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:13.543 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:13.544 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:14.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:14.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:14.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:14.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:14.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:14.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:14.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:14.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:14.125 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:14.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:14.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:14.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:20:14.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:14.702 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:20:15.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:15.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:15.104 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:15.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:15.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:15.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:15.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:16.129 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:16.130 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:16.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:16.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:16.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:20:16.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:16.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:20:17.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:17.521 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:17.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:17.522 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:17.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:17.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:17.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:17.523 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:17.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:17.965 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:17.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:17.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:18.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:18.561 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:18.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:19.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:19.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:19.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:20:19.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:19.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:20:20.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:20.733 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:20.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:20.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:20.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:20.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:20.734 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:20.735 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:21.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:21.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:21.190 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:21.191 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:21.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:21.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:21.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:21.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:21.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:21.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:21.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:21.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:21.776 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:22.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:22.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:22.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:20:22.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:22.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:20:25.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:25.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:25.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:25.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:25.588 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:25.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:25.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:25.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:26.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:26.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:26.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:26.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:26.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:26.656 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:27.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:27.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:27.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:20:27.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:27.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:20:33.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:33.628 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:33.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:33.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:33.628 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:33.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:33.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:33.629 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:34.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:34.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:34.098 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:34.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:34.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:34.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:34.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:34.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:34.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:34.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:34.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:34.692 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:35.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:35.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:35.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:20:35.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:35.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:20:42.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:42.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:42.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:42.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:42.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:42.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:42.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:42.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:43.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:43.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:43.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:43.047 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:43.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:43.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:43.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:43.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:43.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:43.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:43.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:43.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:43.633 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:20:44.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:20:44.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:20:44.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:20:44.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:20:44.809 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:20:44.809 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "61c58565-b225-4d59-b7cc-57d825890622", 2024-09-10T20:20:44.809 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:20:44.809 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:20:44.809 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:20:44.808731+0000", 2024-09-10T20:20:44.809 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.810 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.811 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.812 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:20:44.813 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:17:02.807235Z", 2024-09-10T20:20:44.814 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:17:02.807235Z", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:20:44.815 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:20:44.816 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.817 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 43, 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:17:03.820911+0000", 2024-09-10T20:20:44.818 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:20:00.616318+0000", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:19:58.704136+0000", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:19:59.999735+0000", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.819 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 10, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:20:44.820 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:18:50.314164+0000", 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 16389, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 1, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:20:44.821 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:20:44.822 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "30", 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "pool", 2024-09-10T20:20:44.823 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 1, 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 30, 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [ 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "snapid": 1, 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-09-10T20:19:36.733404+0000", 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap1" 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:20:44.824 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:20:44.825 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "myprofile", 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 8192, 2024-09-10T20:20:44.826 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {} 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "af440182-d88c-48fb-b1e2-3030d54ba58c", 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:20:44.827 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 5, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 37, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 41, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 31, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.828 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:20:44.829 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.830 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.831 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 3163993053 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.832 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/3163993053", 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/3163993053", 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3163993053", 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3163993053", 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "6e593c3e-6dfc-43a0-8d93-a3049d9fe32b", 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:20:44.833 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 25, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.834 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.835 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.836 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:20:44.837 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1972346789", 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1972346789", 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1972346789", 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1972346789", 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:20:44.838 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "8d54aebd-dd27-4eb6-bbb3-ec628062ac61", 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 15, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 29, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 39, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-10T20:20:44.839 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 33, 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.840 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.841 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.842 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 720469603 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.843 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/720469603", 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/720469603", 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/720469603", 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/720469603", 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:20:44.844 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:19:47.936167+0000", 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:17:37.440608+0000", 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:19:48.568082+0000", 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:20:44.845 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:18:12.372788+0000", 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:19:49.158519+0000", 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:18:45.084226+0000", 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.846 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:20:44.847 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "myprofile": { 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-device-class": "", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-num-failure-domains": "0", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-osds-per-failure-domain": "0", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-root": "default", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "1", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:20:44.848 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van", 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: "w": "8" 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:20:44.849 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:20:44.850 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/3163993053,v1:127.0.0.1:6805/3163993053]", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:20:44.851 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:17:10.602403Z", 2024-09-10T20:20:44.852 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/3163993053,v1:127.0.0.1:6803/3163993053]", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3163993053,v1:127.0.0.1:6809/3163993053]", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3163993053,v1:127.0.0.1:6807/3163993053]", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:20:44.853 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.854 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1972346789,v1:127.0.0.1:6813/1972346789]", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:20:44.855 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:20:44.856 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:17:44.846178Z", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1972346789,v1:127.0.0.1:6811/1972346789]", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1972346789,v1:127.0.0.1:6817/1972346789]", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1972346789,v1:127.0.0.1:6815/1972346789]", 2024-09-10T20:20:44.857 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:20:44.858 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/720469603,v1:127.0.0.1:6813/720469603]", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:20:44.859 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:18:18.892162Z", 2024-09-10T20:20:44.860 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/720469603,v1:127.0.0.1:6811/720469603]", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/720469603,v1:127.0.0.1:6817/720469603]", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/720469603,v1:127.0.0.1:6815/720469603]", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:20:44.861 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 0, 2024-09-10T20:20:44.862 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [] 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 37 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 39 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.863 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 43, 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:20:44.864 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:20:44.865 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.866 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.867 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:20:44.868 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:20:44.869 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:20:44.870 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:20:44.871 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:20:44.872 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.873 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.874 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:20:44.875 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 1, 2024-09-10T20:20:44.876 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "test", 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_chooseleaf_tries", 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 5 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_choose_tries", 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 100 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.877 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_indep", 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:20:44.878 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.879 INFO:tasks.workunit.client.0.smithi039.stderr:report 89545843 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:20:44.880 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.881 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:17:03:796977+0000", 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:20:44.882 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.883 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:20:44.884 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:20:44.885 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:20:44.886 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:20:44.887 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:20:44.888 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:20:44.889 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83692, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2164, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489108, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036846592, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2215936, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1550569, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:20:44.890 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:20:44.891 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:20:44.892 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 83692, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2164, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 10, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 81461, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314489108, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322036846592, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2215936, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1550569, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:20:44.893 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 10396, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 83416932 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:20:44.894 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:20:44.895 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:20:44.896 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:20:44.897 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:20:44.898 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1241088, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1241088, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-10T20:20:44.899 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 158913789964 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672993 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 167503724556 2024-09-10T20:20:44.900 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2147483647, 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:20:44.901 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.902 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-10T20:20:44.903 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:19:47.639173+0000", 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 99, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 401, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 551, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:20:44.904 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:20:44.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:20:44.906 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:20:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:20:44.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-10T20:20:44.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:20:44.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:20:44.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2024-09-10T20:20:44.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:20:44.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-10T20:20:44.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-10T20:20:46.551 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-10T20:20:47.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/0 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-10T20:20:47.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-10T20:20:47.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:20:47.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:20:47.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:20:47.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:20:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:20:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:20:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:20:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:20:47.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/0 2024-09-10T20:20:47.092 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:20:47.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:20:47.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:20:47.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/0/whoami 2024-09-10T20:20:47.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:20:47.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:20:47.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:20:47.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:20:47.138 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:47.139+0000 7f5ca48de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:47.140 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:47.143+0000 7f5ca48de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:47.142 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:47.143+0000 7f5ca48de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:20:47.615 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:20:47.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:20:47.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:20:47.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:20:48.122 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 37 up_thru 41 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/3163993053,v1:127.0.0.1:6803/3163993053] [v2:127.0.0.1:6804/3163993053,v1:127.0.0.1:6805/3163993053] exists,up af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:20:48.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:20:48.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:20:48.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:20:48.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=2 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-10T20:20:48.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-10T20:20:48.435 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:48.435+0000 7f5ca48de7c0 -1 Falling back to public interface 2024-09-10T20:20:49.714 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-10T20:20:50.233 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:50.235+0000 7f5ca48de7c0 -1 osd.0 43 log_to_monitors true 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:20:50.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/2 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:20:50.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:20:50.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:20:50.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:20:50.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:20:50.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:20:50.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:20:50.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:20:50.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/2 2024-09-10T20:20:50.269 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:20:50.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:20:50.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:20:50.270 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/2/whoami 2024-09-10T20:20:50.270 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:20:50.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:20:50.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:20:50.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:20:50.314 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:50.315+0000 7f0a4df3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:50.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:50.319+0000 7f0a4df3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:50.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:50.319+0000 7f0a4df3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:20:50.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:20:50.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:20:50.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:20:50.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:20:50.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:20:50.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:20:50.738 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:20:50.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:20:50.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:20:50.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:20:51.242 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/720469603,v1:127.0.0.1:6811/720469603] [v2:127.0.0.1:6812/720469603,v1:127.0.0.1:6813/720469603] exists,up 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:20:51.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:20:51.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:20:51.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:20:51.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-10T20:20:51.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-10T20:20:51.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:20:51.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=1 2024-09-10T20:20:51.245 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:20:51.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-10T20:20:51.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-10T20:20:51.371 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:51.371+0000 7f0a4df3a7c0 -1 Falling back to public interface 2024-09-10T20:20:53.095 INFO:tasks.workunit.client.0.smithi039.stderr:PG '1.0' not found 2024-09-10T20:20:53.122 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:20:53.123+0000 7f0a4df3a7c0 -1 osd.2 43 log_to_monitors true 2024-09-10T20:20:53.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2024-09-10T20:20:53.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-10T20:20:53.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-10T20:20:53.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:20:53.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:20:53.639 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:20:53.639 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 20 .1 2024-09-10T20:20:53.640 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:20:53.640 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:20:53.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:20:53.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:20:53.641 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:20:53.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-10T20:20:53.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:20:53.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:20:53.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:20:53.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:20:53.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:20:54.200 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:20:54.200 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:20:54.200 INFO:tasks.workunit.client.0.smithi039.stderr:2' 2024-09-10T20:20:54.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:20:54.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:20:54.201 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:20:54.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=197568495619 2024-09-10T20:20:54.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 197568495619 2024-09-10T20:20:54.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619' 2024-09-10T20:20:54.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:20:54.376 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:20:54.503 INFO:tasks.workunit.client.0.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T20:20:54.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq= 2024-09-10T20:20:54.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z '' 2024-09-10T20:20:54.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: continue 2024-09-10T20:20:54.505 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:20:54.506 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:20:54.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-10T20:20:54.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-10T20:20:54.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-09-10T20:20:54.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:20:54.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-197568495619 2024-09-10T20:20:54.668 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:20:54.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:20:54.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-197568495619 2024-09-10T20:20:54.670 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:20:54.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=197568495619 2024-09-10T20:20:54.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-09-10T20:20:54.671 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 197568495619 2024-09-10T20:20:54.672 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:20:55.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 197568495619 2024-09-10T20:20:55.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:20:56.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:20:56.168 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:20:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 197568495620 -lt 197568495619 2024-09-10T20:20:56.675 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:20:56.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-214748364803 2024-09-10T20:20:56.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:20:56.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:20:56.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-214748364803 2024-09-10T20:20:56.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:20:56.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-10T20:20:56.680 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 214748364803 2024-09-10T20:20:56.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-10T20:20:56.681 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:20:57.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364803 -lt 214748364803 2024-09-10T20:20:57.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:20:57.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:57.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:57.736 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:57.737 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:58.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:58.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:58.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:58.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:20:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:20:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:20:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:20:58.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:20:58.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:20:58.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:20:58.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:20:58.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:20:58.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:20:58.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:20:58.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:20:58.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:20:59.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:20:59.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:20:59.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:20:59.301 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:20:59.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:20:59.882 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:00.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:00.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:00.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 1 >= 8 )) 2024-09-10T20:21:00.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:00.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.2 2024-09-10T20:21:00.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:00.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:00.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:00.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:00.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:00.695 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:00.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:00.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:01.152 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:01.152 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:01.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:01.153 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:01.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:01.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:01.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:01.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:01.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:01.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:01.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:01.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:01.749 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:02.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 2 >= 8 )) 2024-09-10T20:21:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:02.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.4 2024-09-10T20:21:02.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:02.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:02.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:02.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:02.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:02.740 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:02.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:02.741 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:03.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:03.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:03.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:03.200 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:03.773 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:03.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:03.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:03.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:03.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:03.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:04.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:04.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:04.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 3 >= 8 )) 2024-09-10T20:21:04.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:04.343 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.8 2024-09-10T20:21:05.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:05.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:05.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:05.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:05.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:05.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:05.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:05.147 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:05.594 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:05.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:05.595 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:05.596 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:06.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:06.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:06.216 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:06.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:06.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:06.793 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 4 >= 8 )) 2024-09-10T20:21:06.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:06.794 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 1.6 2024-09-10T20:21:08.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:08.395 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:08.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:08.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:08.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:08.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:08.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:08.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:08.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:08.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:08.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:08.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:09.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:09.434 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 5 >= 8 )) 2024-09-10T20:21:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:10.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 3.2 2024-09-10T20:21:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:13.251 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:13.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:13.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:13.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:13.252 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:13.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:13.253 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:13.710 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:13.710 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:13.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:13.711 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:14.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:14.296 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:14.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:14.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:14.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:14.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 6 >= 8 )) 2024-09-10T20:21:14.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:14.870 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 6.4 2024-09-10T20:21:21.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:21.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:21.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:21.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:21.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:21.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:21.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:21.272 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:21.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:21.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:21.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:21.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:22.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:22.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:22.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:22.375 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 7 >= 8 )) 2024-09-10T20:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:21:22.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 7.3 2024-09-10T20:21:30.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:21:30.267 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:21:30.267 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:21:30.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:21:30.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:21:30.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:21:30.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:21:30.268 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:21:30.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:21:30.722 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:21:30.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:21:30.723 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:21:31.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:21:31.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:21:31.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:21:31.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:21:31.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:21:31.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:21:31.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:21:31.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:21:31.331 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:21:31.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2024-09-10T20:21:31.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2024-09-10T20:21:31.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1655: wait_for_clean: (( 8 >= 8 )) 2024-09-10T20:21:31.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1656: wait_for_clean: ceph report 2024-09-10T20:21:32.521 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:21:32.521 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_fingerprint": "61c58565-b225-4d59-b7cc-57d825890622", 2024-09-10T20:21:32.521 INFO:tasks.workunit.client.0.smithi039.stdout: "version": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:32.521 INFO:tasks.workunit.client.0.smithi039.stdout: "commit": "bad1f558c327094897e4a32e99dd5cb4ae6ccd46", 2024-09-10T20:21:32.521 INFO:tasks.workunit.client.0.smithi039.stdout: "timestamp": "2024-09-10T20:21:32.521225+0000", 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "tag": "", 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "health": { 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "status": "HEALTH_WARN", 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "checks": { 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "OSDMAP_FLAGS": { 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 17 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [], 2024-09-10T20:21:32.522 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "OSD_DOWN": { 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 osds down", 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "osd.1 (root=default,host=smithi039) is down" 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:21:32.523 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "PG_DEGRADED": { 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:21:32.524 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "severity": "HEALTH_WARN", 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "summary": { 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "count": 1 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "detail": [ 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "application not enabled on pool 'test'" 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.525 INFO:tasks.workunit.client.0.smithi039.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "muted": false 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "mutes": [] 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_first_committed": 1, 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap_last_committed": 1, 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "monmap": { 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:17:02.807235Z", 2024-09-10T20:21:32.526 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:17:02.807235Z", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release": 19, 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "min_mon_release_name": "squid", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "election_strategy": 1, 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "disallowed_leaders: ": "", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": false, 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "tiebreaker_mon": "", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_ranks: ": "", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "features": { 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "persistent": [ 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "kraken", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "luminous", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "mimic", 2024-09-10T20:21:32.527 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap-prune", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "nautilus", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "octopus", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "pacific", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "elector-pinging", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "quincy", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "reef", 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "squid" 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.528 INFO:tasks.workunit.client.0.smithi039.stdout: "optional": [] 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "mons": [ 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "rank": 0, 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "a", 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115", 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 0 2024-09-10T20:21:32.529 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:7115/0", 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "priority": 0, 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 0, 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_location": "{}" 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "quorum": [ 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: 0 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.530 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap": { 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 51, 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "fsid": "e3517c37-da15-4e14-a814-7e5f7d5c0537", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "created": "2024-09-10T20:17:03.820911+0000", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:20:54.022699+0000", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "last_up_change": "2024-09-10T20:20:53.709241+0000", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "last_in_change": "2024-09-10T20:19:59.999735+0000", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_num": 5800384, 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_set": [ 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "nodown", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "noin", 2024-09-10T20:21:32.531 INFO:tasks.workunit.client.0.smithi039.stdout: "noout", 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "pglog_hardlimit", 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snapdirs", 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "recovery_deletes", 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "sortbitwise" 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_version": 12, 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "full_ratio": 0.99000000953674316, 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_snapshot": "", 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_max": 1, 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "max_osd": 3, 2024-09-10T20:21:32.532 INFO:tasks.workunit.client.0.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "min_compat_client": "jewel", 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "require_osd_release": "squid", 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "allow_crimson": false, 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "pools": [ 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_name": "test", 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "create_time": "2024-09-10T20:18:50.314164+0000", 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "flags": 16389, 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "size": 3, 2024-09-10T20:21:32.533 INFO:tasks.workunit.client.0.smithi039.stdout: "min_size": 2, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_rule": 1, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "is_stretch_pool": false, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "object_hash": 2, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num": 1, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num": 1, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_target": 1, 2024-09-10T20:21:32.534 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_num_pending": 1, 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pg_merge_meta": { 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "source_pgid": "0.0", 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "ready_epoch": 0, 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_started": 0, 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": 0, 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "source_version": "0'0", 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "target_version": "0'0" 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_change": "30", 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend": "0", 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "auid": 0, 2024-09-10T20:21:32.535 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_mode": "pool", 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_seq": 1, 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_epoch": 30, 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_snaps": [ 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "snapid": 1, 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "stamp": "2024-09-10T20:19:36.733404+0000", 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "snap1" 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps": "[]", 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_bytes": 0, 2024-09-10T20:21:32.536 INFO:tasks.workunit.client.0.smithi039.stdout: "quota_max_objects": 0, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "tiers": [], 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "tier_of": -1, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "read_tier": -1, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "write_tier": -1, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_mode": "none", 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_bytes": 0, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "target_max_objects": 0, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T20:21:32.537 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profile": "myprofile", 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_params": { 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "none" 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_period": 0, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_count": 0, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "use_gmt_hitset": true, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-10T20:21:32.538 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "grade_table": [], 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "stripe_width": 8192, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "expected_num_objects": 0, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "fast_read": false, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "options": {}, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "application_metadata": {} 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "osds": [ 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "af440182-d88c-48fb-b1e2-3030d54ba58c", 2024-09-10T20:21:32.539 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 37, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 43, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 46, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 50, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 45, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.540 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6802", 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6803", 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:21:32.541 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6804", 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6805", 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.542 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6808", 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6809", 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.543 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6806", 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6807", 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1004491106 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.544 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6803/1004491106", 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6805/1004491106", 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1004491106", 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1004491106", 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "6e593c3e-6dfc-43a0-8d93-a3049d9fe32b", 2024-09-10T20:21:32.545 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 0, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 0, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 0, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 10, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 25, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 32, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.546 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:21:32.547 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:21:32.548 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.549 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 1972346789 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.550 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/1972346789", 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/1972346789", 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1972346789", 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1972346789", 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "exists" 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "uuid": "8d54aebd-dd27-4eb6-bbb3-ec628062ac61", 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 1, 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "in": 1, 2024-09-10T20:21:32.551 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 1, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_affinity": 1, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_begin": 39, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "last_clean_end": 43, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 50, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "up_thru": 0, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "down_at": 49, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "lost_at": 0, 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addrs": { 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6810", 2024-09-10T20:21:32.552 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6811", 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addrs": { 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6812", 2024-09-10T20:21:32.553 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6813", 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.554 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6816", 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6817", 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "addrvec": [ 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v2", 2024-09-10T20:21:32.555 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6814", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "v1", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "addr": "127.0.0.1:6815", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "nonce": 425664931 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "public_addr": "127.0.0.1:6811/425664931", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "cluster_addr": "127.0.0.1:6813/425664931", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_back_addr": "127.0.0.1:6817/425664931", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "heartbeat_front_addr": "127.0.0.1:6815/425664931", 2024-09-10T20:21:32.556 INFO:tasks.workunit.client.0.smithi039.stdout: "state": [ 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "exists", 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "up" 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_xinfo": [ 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:20:50.512999+0000", 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:21:32.557 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:17:37.440608+0000", 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:19:48.568082+0000", 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:18:12.372788+0000", 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.558 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "down_stamp": "2024-09-10T20:20:53.466616+0000", 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_probability": 0, 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "laggy_interval": 0, 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "features": 4540719139924082687, 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "old_weight": 0, 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-10T20:18:45.084226+0000", 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "dead_epoch": 0 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap": [], 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_items": [], 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-10T20:21:32.559 INFO:tasks.workunit.client.0.smithi039.stdout: "pg_temp": [], 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "primary_temp": [], 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "blocklist": {}, 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "range_blocklist": {}, 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "erasure_code_profiles": { 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "default": { 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "2", 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van" 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "myprofile": { 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-device-class": "", 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-10T20:21:32.560 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-num-failure-domains": "0", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-osds-per-failure-domain": "0", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "crush-root": "default", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "k": "2", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "m": "1", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "plugin": "jerasure", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "technique": "reed_sol_van", 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "w": "8" 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "removed_snaps_queue": [], 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "new_removed_snaps": [], 2024-09-10T20:21:32.561 INFO:tasks.workunit.client.0.smithi039.stdout: "new_purged_snaps": [], 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "crush_node_flags": {}, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "device_class_flags": {}, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode": { 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_metadata": [ 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:21:32.562 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6804/1004491106,v1:127.0.0.1:6805/1004491106]", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-10T20:21:32.563 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:17:10.602403Z", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:21:32.564 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6802/1004491106,v1:127.0.0.1:6803/1004491106]", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1004491106,v1:127.0.0.1:6809/1004491106]", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1004491106,v1:127.0.0.1:6807/1004491106]", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:21:32.565 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/0", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/1972346789,v1:127.0.0.1:6813/1972346789]", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:21:32.566 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:21:32.567 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:17:44.846178Z", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:21:32.568 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/1972346789,v1:127.0.0.1:6811/1972346789]", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1972346789,v1:127.0.0.1:6817/1972346789]", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1972346789,v1:127.0.0.1:6815/1972346789]", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:21:32.569 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/1", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "arch": "x86_64", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "back_addr": "[v2:127.0.0.1:6812/425664931,v1:127.0.0.1:6813/425664931]", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "back_iface": "", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs": "1", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_db": "0", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_dedicated_wal": "0", 2024-09-10T20:21:32.570 INFO:tasks.workunit.client.0.smithi039.stdout: "bluefs_single_shared_device": "1", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_allocation_from_file": "0", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_access_mode": "file", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_block_size": "4096", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_devices": "sda", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_rotational": "1", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_size": "107374182400", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_support_discard": "0", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_bdev_type": "hdd", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "bluestore_min_alloc_size": "4096", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_release": "squid", 2024-09-10T20:21:32.571 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_short": "19.3.0-4858-gbad1f558", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "ceph_version_when_created": "ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev)", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "created_at": "2024-09-10T20:18:18.892162Z", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "default_device_class": "hdd", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4HQ5B", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": "sda", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "distro": "ubuntu", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "distro_version": "22.04", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "front_addr": "[v2:127.0.0.1:6810/425664931,v1:127.0.0.1:6811/425664931]", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "front_iface": "", 2024-09-10T20:21:32.572 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/425664931,v1:127.0.0.1:6817/425664931]", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/425664931,v1:127.0.0.1:6815/425664931]", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "hostname": "smithi039", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "journal_rotational": "1", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_description": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "kernel_version": "5.15.0-119-generic", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_swap_kb": "2097148", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "mem_total_kb": "32746856", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "os": "Linux", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_data": "td/divergent-priors/2", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "osdspec_affinity": "", 2024-09-10T20:21:32.573 INFO:tasks.workunit.client.0.smithi039.stdout: "rotational": "1" 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_clean_epochs": { 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "min_last_epoch_clean": 26, 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "last_epoch_clean": { 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "per_pool": [ 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "floor": 26 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_epochs": [ 2024-09-10T20:21:32.574 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 47 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 50 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_first_committed": 1, 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "osdmap_last_committed": 51, 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "crushmap": { 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: "devices": [ 2024-09-10T20:21:32.575 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.0", 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.1", 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd.2", 2024-09-10T20:21:32.576 INFO:tasks.workunit.client.0.smithi039.stdout: "class": "hdd" 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "types": [ 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 0, 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "osd" 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "host" 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 2, 2024-09-10T20:21:32.577 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "chassis" 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 3, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "rack" 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 4, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "row" 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 5, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pdu" 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.578 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 6, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "pod" 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 7, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "room" 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 8, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "datacenter" 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 9, 2024-09-10T20:21:32.579 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "zone" 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 10, 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "region" 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "root" 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "buckets": [ 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -1, 2024-09-10T20:21:32.580 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default", 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.581 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -2, 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "default~hdd", 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 11, 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "root", 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:21:32.582 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.583 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.583 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -3, 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039", 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:21:32.584 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:21:32.585 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "id": -4, 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "name": "smithi039~hdd", 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "type_id": 1, 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "type_name": "host", 2024-09-10T20:21:32.586 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 19206, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "alg": "straw2", 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "hash": "rjenkins1", 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "items": [ 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 0, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 0 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 1, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 1 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.587 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "id": 2, 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "weight": 6402, 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "pos": 2 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "rules": [ 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 0, 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "replicated_rule", 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 1, 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:21:32.588 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_firstn", 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.589 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_id": 1, 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "rule_name": "test", 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "type": 3, 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "steps": [ 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_chooseleaf_tries", 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 5 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "set_choose_tries", 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 100 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.590 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "take", 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "item": -1, 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "item_name": "default" 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "choose_indep", 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 0, 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "type": "osd" 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: "op": "emit" 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.591 INFO:tasks.workunit.client.0.smithi039.stdout: ] 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "tunables": { 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_tries": 0, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_local_fallback_tries": 0, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_total_tries": 50, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_descend_once": 1, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_vary_r": 1, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "chooseleaf_stable": 1, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_descents": 100, 2024-09-10T20:21:32.592 INFO:tasks.workunit.client.0.smithi039.stdout: "msr_collision_tries": 100, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "straw_calc_version": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "allowed_bucket_algs": 54, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "profile": "jewel", 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "optimal_tunables": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "legacy_tunables": 0, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "minimum_required_version": "jewel", 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables2": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v2_rules": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables3": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v3_rules": 0, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v4_buckets": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "require_feature_tunables5": 1, 2024-09-10T20:21:32.593 INFO:tasks.workunit.client.0.smithi039.stdout: "has_v5_rules": 0, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "has_msr_rules": 0 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "choose_args": {} 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "fsmap": { 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 1, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "btime": "2024-09-10T20:17:03:796977+0000", 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "default_fscid": -1, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": { 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "compat": {}, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "ro_compat": {}, 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "incompat": { 2024-09-10T20:21:32.594 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_1": "base v0.20", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_2": "client writeable ranges", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_3": "default file layouts on dirs", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_4": "dir inode in separate object", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_5": "mds uses versioned encoding", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_8": "no anchor table", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_9": "file layout v2", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_10": "snaprealm v2", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_11": "minor log segments", 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_12": "quiesce subvolumes" 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.595 INFO:tasks.workunit.client.0.smithi039.stdout: "feature_flags": { 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "enable_multiple": true, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "ever_enabled_multiple": true 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "standbys": [], 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "filesystems": [] 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_first_committed": 1, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "mdsmap_last_committed": 1, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "auth": { 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 5, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "num_secrets": 3 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.596 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_active": 1, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_unknown": 0, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osd": 3, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_sum": { 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:21:32.597 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:21:32.598 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:21:32.599 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:21:32.600 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 0 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum": { 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:21:32.601 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 85120, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2292, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 15, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 82736, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314487680, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035384320, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2347008, 2024-09-10T20:21:32.602 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1605477, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 16263, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 84721785 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:21:32.603 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_sum_by_class": { 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "hdd": { 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "up_from": 0, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 0, 2024-09-10T20:21:32.604 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pgs": 3, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_osds": 3, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_osds": 3, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "num_per_pool_omap_osds": 3, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "kb": 314572800, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used": 85120, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_data": 2292, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_omap": 15, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_used_meta": 82736, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "kb_avail": 314487680, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "statfs": { 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 322122547200, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 322035384320, 2024-09-10T20:21:32.605 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 2347008, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1605477, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 16263, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 84721785 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "hb_peers": [], 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "snap_trim_queue_len": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_snap_trimming": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shards_repaired": 0, 2024-09-10T20:21:32.606 INFO:tasks.workunit.client.0.smithi039.stdout: "op_queue_age_hist": { 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "histogram": [], 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "upper_bound": 1 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "perf_stat": { 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ms": 0, 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ms": 0, 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "commit_latency_ns": 0, 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "apply_latency_ns": 0 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "alerts": [], 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "network_ping_times": [] 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.607 INFO:tasks.workunit.client.0.smithi039.stdout: "pool_stats": [ 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "poolid": 1, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg": 1, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "stat_sum": { 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes": 62667, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects": 101, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_clones": 1, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_object_copies": 303, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing_on_primary": 0, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_missing": 0, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_degraded": 101, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_misplaced": 0, 2024-09-10T20:21:32.608 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_unfound": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_dirty": 101, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_whiteouts": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_read_kb": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write": 101, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_write_kb": 101, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_scrub_errors": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_shallow_scrub_errors": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_deep_scrub_errors": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_recovered": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_recovered": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_keys_recovered": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_omap": 0, 2024-09-10T20:21:32.609 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_hit_set_archive": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_bytes_hit_set_archive": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_kb": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_kb": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_promote": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_high": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_flush_mode_low": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_some": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_evict_mode_full": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_pinned": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_legacy_snapsets": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_large_omap_objects": 0, 2024-09-10T20:21:32.610 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_manifest": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_bytes": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "num_omap_keys": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "num_objects_repaired": 0 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "store_stats": { 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "total": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "available": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "internally_reserved": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "allocated": 1241088, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "data_stored": 1241088, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_allocated": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "data_compressed_original": 0, 2024-09-10T20:21:32.611 INFO:tasks.workunit.client.0.smithi039.stdout: "omap_allocated": 0, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "internal_metadata": 0 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "log_size": 18, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "ondisk_log_size": 18, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "up": 3, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "acting": 3, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "num_store_stats": 3 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "osd_stats": [ 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:21:32.612 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 197568495627 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 42949672993 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "seq": 214748364810 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_state": [ 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "state": "active+undersized+degraded", 2024-09-10T20:21:32.613 INFO:tasks.workunit.client.0.smithi039.stdout: "num": 1 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_pg_by_osd": [ 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2147483647, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 2, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:21:32.614 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 0, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 1, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 1, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "osd": 1, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "num_primary_pg": 0, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "num_acting_pg": 0, 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: "num_up_not_acting_pg": 0 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.615 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": [ 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: { 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "pool": 1, 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "purged_snaps": {} 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: ], 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "servicemap": { 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "epoch": 5, 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "modified": "2024-09-10T20:19:47.639173+0000", 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "services": {} 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: }, 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_first_committed": 1, 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "mgrstat_last_committed": 123, 2024-09-10T20:21:32.616 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_first_committed": 1, 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: "logm_last_committed": 480, 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: "paxos": { 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: "first_committed": 1, 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: "last_committed": 661, 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: "last_pn": 0, 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: "accepted_pn": 0 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout: } 2024-09-10T20:21:32.617 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:21:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:report 1001471731 2024-09-10T20:21:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: return 1 2024-09-10T20:21:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-10T20:21:32.618 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery 2024-09-10T20:21:32.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:32.948 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:32.948 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-10T20:20:55.231838+0000 43'102 51:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:07:22.536809+0000 0 0 2024-09-10T20:21:32.948 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:32.948 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:32.948 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:32.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:21:32.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-10T20:21:32.966 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T20:21:32.967 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T20:21:32.967 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:21:32.967 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:32.967 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:21:32.968 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.0.asok 2024-09-10T20:21:32.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-10T20:21:32.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.0.asok set_recovery_delay 100000 2024-09-10T20:21:33.077 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:21:33.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-10T20:21:33.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-10T20:21:33.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.2 2024-09-10T20:21:33.092 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.2 ']' 2024-09-10T20:21:33.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:21:33.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:33.092 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:21:33.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.2.asok 2024-09-10T20:21:33.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-10T20:21:33.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.2.asok set_recovery_delay 100000 2024-09-10T20:21:33.211 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:21:33.224 INFO:tasks.workunit.client.0.smithi039.stdout:revive divergent 1 2024-09-10T20:21:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-10T20:21:33.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:33.668 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:33.668 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-10T20:20:55.231838+0000 43'102 51:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:07:22.536809+0000 0 0 2024-09-10T20:21:33.668 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:33.668 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:33.668 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:33.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-10T20:21:34.378 INFO:tasks.workunit.client.0.smithi039.stderr:noup is set 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:21:34.401 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:21:34.402 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:21:34.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:21:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:21:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:21:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:21:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:21:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:21:34.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:21:34.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:21:34.406 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:21:34.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:21:34.407 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:21:34.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:21:34.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:21:34.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:21:34.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:21:34.443 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:34.443+0000 7f3d28bb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:34.445 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:34.447+0000 7f3d28bb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:34.447 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:34.447+0000 7f3d28bb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:34.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: return 0 2024-09-10T20:21:34.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-10T20:21:35.694 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:35.695+0000 7f3d28bb07c0 -1 Falling back to public interface 2024-09-10T20:21:37.151 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:37.151+0000 7f3d28bb07c0 -1 osd.1 30 log_to_monitors true 2024-09-10T20:21:39.885 INFO:tasks.workunit.client.0.smithi039.stdout:delay recovery divergent 2024-09-10T20:21:39.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-10T20:21:39.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:40.324 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:40.325 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-10T20:20:55.231838+0000 43'102 53:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:07:22.536809+0000 0 0 2024-09-10T20:21:40.325 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:40.325 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:40.326 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:40.342 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-10T20:21:40.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T20:21:40.343 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T20:21:40.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:21:40.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:40.344 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:21:40.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27146/ceph-osd.1.asok 2024-09-10T20:21:40.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-10T20:21:40.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27146/ceph-osd.1.asok set_recovery_delay 100000 2024-09-10T20:21:40.461 INFO:tasks.workunit.client.0.smithi039.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-10T20:21:40.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-10T20:21:41.306 INFO:tasks.workunit.client.0.smithi039.stderr:noup is unset 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:41.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:41.331 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:21:41.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:41.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:21:41.948 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/1004491106,v1:127.0.0.1:6803/1004491106] [v2:127.0.0.1:6804/1004491106,v1:127.0.0.1:6805/1004491106] exists,up af440182-d88c-48fb-b1e2-3030d54ba58c 2024-09-10T20:21:41.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:41.950 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:41.951 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:21:41.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:21:41.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:42.417 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/735440740,v1:127.0.0.1:6819/735440740] [v2:127.0.0.1:6820/735440740,v1:127.0.0.1:6821/735440740] exists,up 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:42.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:42.420 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:21:42.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:42.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:21:42.941 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/425664931,v1:127.0.0.1:6811/425664931] [v2:127.0.0.1:6812/425664931,v1:127.0.0.1:6813/425664931] exists,up 8d54aebd-dd27-4eb6-bbb3-ec628062ac61 2024-09-10T20:21:42.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:42.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:42.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:42.942 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:43.362 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:43.362 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-09-10T20:20:55.231838+0000 43'102 54:147 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:07:22.536809+0000 0 0 2024-09-10T20:21:43.362 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:43.362 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:43.362 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:43.377 INFO:tasks.workunit.client.0.smithi039.stdout:wait for peering 2024-09-10T20:21:43.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-10T20:21:43.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:43.806 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:43.806 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 activating 2024-09-10T20:21:42.319561+0000 43'102 57:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:21:43.806 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:43.807 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:43.807 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:43.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-10T20:21:43.970 INFO:tasks.workunit.client.0.smithi039.stdout:killing divergent 1 2024-09-10T20:21:43.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-10T20:21:43.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:44.401 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:44.401 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 activating 2024-09-10T20:21:42.319561+0000 43'102 57:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:21:44.402 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:44.402 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:44.402 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:44.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-10T20:21:44.414 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:21:44.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:21:44.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:21:44.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:21:44.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:21:44.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:21:44.523 INFO:tasks.workunit.client.0.smithi039.stdout:reviving divergent 1 2024-09-10T20:21:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-10T20:21:44.524 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:21:44.952 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:21:44.952 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 activating 2024-09-10T20:21:42.319561+0000 43'102 57:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 no scrub is scheduled 0 0 2024-09-10T20:21:44.952 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:21:44.952 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:21:44.952 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:21:44.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-10T20:21:44.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/divergent-priors 2024-09-10T20:21:44.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/divergent-priors/1 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-10T20:21:44.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:21:44.972 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:21:44.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/divergent-priors/1 2024-09-10T20:21:44.974 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:21:44.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:21:44.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=e3517c37-da15-4e14-a814-7e5f7d5c0537 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27146/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:21:44.975 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/divergent-priors/1/whoami 2024-09-10T20:21:44.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:21:44.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:21:44.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:21:44.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:21:45.013 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:45.015+0000 7f42269937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:45.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:45.015+0000 7f42269937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:45.017 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:45.019+0000 7f42269937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:21:45.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:21:45.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:21:45.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:21:45.464 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:21:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:21:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:21:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:21:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:21:45.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:21:45.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:21:45.923 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/735440740,v1:127.0.0.1:6819/735440740] [v2:127.0.0.1:6820/735440740,v1:127.0.0.1:6821/735440740] exists,up 6e593c3e-6dfc-43a0-8d93-a3049d9fe32b 2024-09-10T20:21:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:21:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:21:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:21:45.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-10T20:21:46.102 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:46.102+0000 7f42269937c0 -1 Falling back to public interface 2024-09-10T20:21:48.019 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:21:48.018+0000 7f42269937c0 -1 osd.1 57 log_to_monitors true 2024-09-10T20:22:05.925 INFO:tasks.workunit.client.0.smithi039.stdout:allowing recovery 2024-09-10T20:22:05.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-10T20:22:05.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:22:06.365 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:22:06.365 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-10T20:21:49.961383+0000 57'103 62:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:05:05.358962+0000 0 0 2024-09-10T20:22:06.365 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:22:06.365 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:22:06.366 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:22:06.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-10T20:22:06.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-10T20:22:06.543 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:22:06.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-10T20:22:06.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-10T20:22:06.700 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:22:06.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-10T20:22:06.714 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-10T20:22:06.859 INFO:tasks.workunit.client.0.smithi039.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:22:06.873 INFO:tasks.workunit.client.0.smithi039.stdout:reading divergent objects 2024-09-10T20:22:06.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-10T20:22:06.873 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-10T20:22:07.292 INFO:tasks.workunit.client.0.smithi039.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T20:22:07.293 INFO:tasks.workunit.client.0.smithi039.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-09-10T20:21:49.961383+0000 57'103 62:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-10T20:18:50.378948+0000 0'0 2024-09-10T20:18:50.378948+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:05:05.358962+0000 0 0 2024-09-10T20:22:07.293 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-09-10T20:22:07.293 INFO:tasks.workunit.client.0.smithi039.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T20:22:07.293 INFO:tasks.workunit.client.0.smithi039.stderr:dumped pgs 2024-09-10T20:22:07.311 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-10T20:22:07.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-10T20:22:07.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:22:07.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-10T20:22:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:22:07.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-10T20:22:07.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:22:07.390 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-10T20:22:07.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-10T20:22:07.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-10T20:22:07.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-10T20:22:07.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-10T20:22:07.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:22:07.468 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-10T20:19:28.504671+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-10T20:19:28.504671+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4645.0:1 2024-09-10T20:19:36.969295+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4645.0:1 2024-09-10T20:19:36.969295+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:22:07.469 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-09-10T20:22:07.470 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-10T20:22:07.470 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:22:07.470 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-09-10T20:22:07.470 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4645.0:1 2024-09-10T20:19:36.969295+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-09-10T20:22:07.470 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-10T20:22:07.471 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4636.0:1 2024-09-10T20:19:35.729649+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4642.0:1 2024-09-10T20:19:36.734517+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:22:07.471 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4636.0:1 2024-09-10T20:19:35.729649+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:22:07.471 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4642.0:1 2024-09-10T20:19:36.734517+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:22:07.471 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-10T20:22:07.472 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-09-10T20:22:07.472 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-10T20:22:07.472 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-10T20:22:07.472 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-09-10T20:22:07.472 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4642.0:1 2024-09-10T20:19:36.734517+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-09-10T20:22:07.472 INFO:tasks.workunit.client.0.smithi039.stdout:td/divergent-priors/osd.1.log:2024-09-10T20:21:42.315+0000 7f3d0d33b640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-10T20:22:07.473 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-10T20:22:07.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-09-10T20:22:07.478 INFO:tasks.workunit.client.0.smithi039.stdout:success 2024-09-10T20:22:07.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-10T20:22:07.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-10T20:22:07.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-10T20:22:07.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-10T20:22:08.105 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'test' does not exist 2024-09-10T20:22:08.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-10T20:22:08.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:08.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:08.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:08.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:08.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-10T20:22:08.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:22:08.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:22:08.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:22:08.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:08.437 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:08.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:08.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:08.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:08.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:08.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:22:08.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:22:08.446 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:22:08.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:22:08.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:22:08.447 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:22:08.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:08.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:22:08.448 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:22:08.449 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:08.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:22:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:22:08.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:22:08.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:22:08.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:08.475 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:22:08.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/divergent-priors 0 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/divergent-priors 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/divergent-priors KILL 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:08.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:08.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:08.479 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:22:08.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:22:08.480 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:22:08.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:22:08.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:22:08.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:22:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:08.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:22:08.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:22:08.483 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:08.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:22:08.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:22:08.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/divergent-priors 2024-09-10T20:22:08.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:22:08.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:08.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27146 2024-09-10T20:22:08.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27146 2024-09-10T20:22:08.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:22:08.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:22:08.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:22:08.488 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:22:08.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:22:08.501 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-10T20:22:08.501 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-10T20:22:08.552 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:22:08.558 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:22:08.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:22:08.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:22:08.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:22:08.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:22:08.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/ec-error-rollforward 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none ' 2024-09-10T20:22:08.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-10T20:22:08.560 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:22:08.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ec-error-rollforward 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ec-error-rollforward 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:22:08.562 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:22:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:22:08.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:22:08.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:22:08.563 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:22:08.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:22:08.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:22:08.564 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:22:08.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:08.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:22:08.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:22:08.565 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:22:08.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:22:08.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:22:08.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-10T20:22:08.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:22:08.568 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:08.569 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:08.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.83789 2024-09-10T20:22:08.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:22:08.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:22:08.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ec-error-rollforward 2024-09-10T20:22:08.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:22:08.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:08.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:08.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.83789 2024-09-10T20:22:08.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:22:08.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:22:08.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:22:08.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ec-error-rollforward 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ec-error-rollforward/a 2024-09-10T20:22:08.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-10T20:22:09.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:22:09.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:09.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:09.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:09.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:09.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:09.051 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:22:09.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:22:09.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:22:09.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:22:09.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:22:09.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:22:09.264 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:22:09.265 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:22:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:22:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:22:09.265 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:22:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:22:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:09.266 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.83789/ceph-mon.a.asok 2024-09-10T20:22:09.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:22:09.266 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.83789/ceph-mon.a.asok config get fsid 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:22:09.382 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:22:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:22:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:09.383 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:09.384 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.83789/ceph-mon.a.asok 2024-09-10T20:22:09.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:22:09.384 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.83789/ceph-mon.a.asok config get mon_host 2024-09-10T20:22:09.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-10T20:22:09.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ec-error-rollforward 2024-09-10T20:22:09.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:22:09.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:22:09.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:22:09.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ec-error-rollforward/x 2024-09-10T20:22:09.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:22:10.396 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:22:10.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:10.397 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:10.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:10.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:10.397 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:10.398 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:22:10.398 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:22:10.400 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:22:10.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-10T20:22:10.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-10T20:22:10.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:22:10.460 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:22:10.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:22:10.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d1d3b79b-e313-4d4a-b884-878c9ccb1d17 2024-09-10T20:22:10.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d1d3b79b-e313-4d4a-b884-878c9ccb1d17' 2024-09-10T20:22:10.463 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 d1d3b79b-e313-4d4a-b884-878c9ccb1d17 2024-09-10T20:22:10.463 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:22:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQByquBmRhm9HBAAQehXSBphPJdyR6yuatbbNg== 2024-09-10T20:22:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQByquBmRhm9HBAAQehXSBphPJdyR6yuatbbNg=="}' 2024-09-10T20:22:10.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d1d3b79b-e313-4d4a-b884-878c9ccb1d17 -i td/ec-error-rollforward/0/new.json 2024-09-10T20:22:10.764 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:22:10.779 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-10T20:22:10.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByquBmRhm9HBAAQehXSBphPJdyR6yuatbbNg== --osd-uuid d1d3b79b-e313-4d4a-b884-878c9ccb1d17 2024-09-10T20:22:10.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:10.814+0000 7f91c7e187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:10.833 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:10.834+0000 7f91c7e187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:10.834 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:10.834+0000 7f91c7e187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:10.835 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:10.834+0000 7f91c7e187c0 -1 bdev(0x55c262f4bc00 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:22:10.835 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:10.834+0000 7f91c7e187c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-10T20:22:17.003 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-10T20:22:17.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:22:17.005 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:22:17.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:22:17.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:22:17.619 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:22:17.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:22:17.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:22:17.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:22:17.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:22:17.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:22:17.650 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:17.650+0000 7fb071af57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:17.652 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:17.654+0000 7fb071af57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:17.653 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:17.654+0000 7fb071af57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:18.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:22:18.084 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:22:18.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:18.084 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:18.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:18.922 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:18.922+0000 7fb071af57c0 -1 Falling back to public interface 2024-09-10T20:22:19.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:19.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:19.560 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:22:19.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:22:19.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:19.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:20.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:20.644 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:20.646+0000 7fb071af57c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:22:21.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:21.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:21.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:22:21.030 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:22:21.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:21.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:21.638 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:22.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:22.640 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:22.641 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:22:22.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:22:22.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:22.641 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:23.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:24.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:24.113 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:22:24.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:22:24.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:24.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:24.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:25.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:25.728 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:22:25.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:22:25.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:25.729 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:27.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:27.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:27.195 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:22:27.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:22:27.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:27.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:27.629 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:28.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:28.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:28.632 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:22:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:22:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:28.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:29.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:30.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:30.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:30.074 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:22:30.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:22:30.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:30.075 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:30.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:31.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:31.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:31.526 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:22:31.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:22:31.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:31.526 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:31.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:32.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:32.978 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:22:32.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:22:32.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:32.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:33.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:34.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:34.791 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:34.791 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:22:34.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:22:34.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:34.792 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:35.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:36.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:36.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:36.546 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:22:36.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:22:36.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:36.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:37.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:38.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:38.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:38.589 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:22:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:22:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:38.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:39.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:40.567 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:40.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:40.568 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:22:40.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:22:40.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:40.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:41.624 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:42.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:42.626 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:42.626 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:22:42.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:22:42.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:42.627 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:22:43.144 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2355015296,v1:127.0.0.1:6803/2355015296] [v2:127.0.0.1:6804/2355015296,v1:127.0.0.1:6805/2355015296] exists,up d1d3b79b-e313-4d4a-b884-878c9ccb1d17 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:22:43.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:22:43.148 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:22:43.149 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82 2024-09-10T20:22:43.150 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82' 2024-09-10T20:22:43.151 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82 2024-09-10T20:22:43.151 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:22:43.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCTquBm6x5SChAAx1P3tQot+3qY00qQhKtVog== 2024-09-10T20:22:43.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCTquBm6x5SChAAx1P3tQot+3qY00qQhKtVog=="}' 2024-09-10T20:22:43.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82 -i td/ec-error-rollforward/1/new.json 2024-09-10T20:22:43.730 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:22:43.750 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-10T20:22:43.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTquBm6x5SChAAx1P3tQot+3qY00qQhKtVog== --osd-uuid 3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82 2024-09-10T20:22:43.785 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:43.785+0000 7fb5e3a957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:43.811 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:43.809+0000 7fb5e3a957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:43.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:43.813+0000 7fb5e3a957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:43.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:43.813+0000 7fb5e3a957c0 -1 bdev(0x5605ebd51c00 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:22:43.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:43.813+0000 7fb5e3a957c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-10T20:22:49.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-10T20:22:49.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:22:49.381 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:22:49.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:22:49.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:22:49.981 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:22:49.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:22:49.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:22:49.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:22:49.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:22:49.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:22:50.017 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:50.017+0000 7f10012a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:50.020 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:50.021+0000 7f10012a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:50.021 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:50.021+0000 7f10012a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:22:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:22:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:22:50.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:22:50.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:22:50.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:22:50.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:50.446 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:22:50.446 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:22:50.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:50.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:50.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:51.029 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:51.029+0000 7f10012a07c0 -1 Falling back to public interface 2024-09-10T20:22:51.938 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:51.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:51.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:22:51.939 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:22:51.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:51.939 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:52.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:52.984 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:22:52.981+0000 7f10012a07c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:22:53.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:53.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:53.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:22:53.433 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:22:53.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:53.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:54.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:55.072 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:55.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:55.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:22:55.073 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:22:55.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:55.073 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:55.520 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:56.522 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:56.523 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:22:56.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:22:56.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:56.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:57.067 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:58.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:58.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:58.069 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:22:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:22:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:58.070 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:22:58.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:22:59.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:22:59.527 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:22:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:22:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:22:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:22:59.528 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:00.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:01.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:01.021 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:23:01.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:01.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:23:01.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:01.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:01.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:02.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:02.467 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:02.467 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:23:02.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:23:02.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:02.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:02.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:03.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:03.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:03.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:23:03.935 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:23:03.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:03.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:04.409 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:05.410 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:05.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:05.411 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:23:05.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:23:05.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:05.411 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:06.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:07.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:23:07.299 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:23:07.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:07.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:08.063 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:09.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:09.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:09.066 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:23:09.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:23:09.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:09.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:09.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:10.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:10.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:10.852 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:23:10.852 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:23:10.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:10.853 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:11.801 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:12.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:12.803 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:12.804 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:23:12.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:23:12.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:12.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:13.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:14.620 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:14.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:14.621 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:23:14.621 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:23:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:14.622 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:15.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:16.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:16.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:16.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:23:16.266 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:23:16.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:16.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:23:16.769 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/683245533,v1:127.0.0.1:6811/683245533] [v2:127.0.0.1:6812/683245533,v1:127.0.0.1:6813/683245533] exists,up 3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:23:16.771 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:23:16.772 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:23:16.774 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:23:16.775 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-10T20:23:16.785 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:23:16.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d5f25bd3-76b8-4ffc-b421-f17d94966302 2024-09-10T20:23:16.787 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d5f25bd3-76b8-4ffc-b421-f17d94966302' 2024-09-10T20:23:16.787 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 d5f25bd3-76b8-4ffc-b421-f17d94966302 2024-09-10T20:23:16.787 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:23:16.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC0quBmvEMZMBAAa9lZ4MK/EpW/fguxC8XdOg== 2024-09-10T20:23:16.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC0quBmvEMZMBAAa9lZ4MK/EpW/fguxC8XdOg=="}' 2024-09-10T20:23:16.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d5f25bd3-76b8-4ffc-b421-f17d94966302 -i td/ec-error-rollforward/2/new.json 2024-09-10T20:23:17.320 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:23:17.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-10T20:23:17.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0quBmvEMZMBAAa9lZ4MK/EpW/fguxC8XdOg== --osd-uuid d5f25bd3-76b8-4ffc-b421-f17d94966302 2024-09-10T20:23:17.371 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:17.368+0000 7f44ef54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:17.373 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:17.372+0000 7f44ef54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:17.372+0000 7f44ef54f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:17.372+0000 7f44ef54f7c0 -1 bdev(0x55ce18199c00 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:23:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:17.372+0000 7f44ef54f7c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-10T20:23:23.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-10T20:23:23.292 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:23:23.292 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:23:23.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:23:23.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:23:23.893 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:23:23.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:23:23.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:23:23.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:23:23.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:23:23.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:23:23.924 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:23.924+0000 7fae247397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:23.962 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:23.960+0000 7fae247397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:23.963 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:23.960+0000 7fae247397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:24.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:23:24.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:23:24.355 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:23:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:23:24.356 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:23:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:23:24.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:23:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:24.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:24.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:25.451 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:25.448+0000 7fae247397c0 -1 Falling back to public interface 2024-09-10T20:23:25.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:25.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:25.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:23:25.842 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:23:25.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:25.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:26.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:26.884 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:26.884+0000 7fae247397c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:23:27.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:27.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:27.325 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:23:27.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:23:27.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:27.326 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:28.033 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:29.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:29.035 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:23:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:23:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:29.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:29.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:30.482 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:30.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:30.483 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:23:30.483 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:23:30.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:30.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:30.928 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:31.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:31.930 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:23:31.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:31.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:23:31.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:31.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:32.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:33.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:33.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:33.405 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:23:33.406 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:23:33.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:33.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:34.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:35.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:35.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:35.102 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:23:35.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:23:35.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:35.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:35.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:36.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:36.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:36.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:23:36.706 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:23:36.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:36.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:37.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:38.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:38.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:38.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:23:38.144 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:23:38.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:38.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:38.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:39.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:39.648 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:39.648 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:23:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:23:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:39.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:40.425 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:41.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:41.427 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:23:41.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:23:41.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:41.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:42.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:43.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:43.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:43.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:23:43.560 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:23:43.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:43.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:44.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:45.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:45.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:45.020 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:23:45.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:23:45.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:45.021 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:45.699 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:46.700 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:46.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:46.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:23:46.701 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:23:46.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:46.701 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:47.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:48.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:48.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:48.173 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:23:48.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:23:48.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:48.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:48.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:49.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:49.858 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:23:49.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:49.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:23:49.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:49.859 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:23:50.329 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/672733083,v1:127.0.0.1:6819/672733083] [v2:127.0.0.1:6820/672733083,v1:127.0.0.1:6821/672733083] exists,up d5f25bd3-76b8-4ffc-b421-f17d94966302 2024-09-10T20:23:50.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:23:50.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:23:50.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:23:50.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-10T20:23:50.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ec-error-rollforward 2024-09-10T20:23:50.332 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:23:50.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:23:50.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:23:50.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:23:50.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:23:50.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611 2024-09-10T20:23:50.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611' 2024-09-10T20:23:50.337 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611 2024-09-10T20:23:50.337 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:23:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDWquBmtoZWFRAAZv2Y4+UWOG9GnbZO6PoaxA== 2024-09-10T20:23:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDWquBmtoZWFRAAZv2Y4+UWOG9GnbZO6PoaxA=="}' 2024-09-10T20:23:50.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611 -i td/ec-error-rollforward/3/new.json 2024-09-10T20:23:50.913 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:23:50.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-10T20:23:50.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWquBmtoZWFRAAZv2Y4+UWOG9GnbZO6PoaxA== --osd-uuid 3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611 2024-09-10T20:23:50.964 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:50.963+0000 7ff89fb557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:50.966 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:50.967+0000 7ff89fb557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:50.967 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:50.967+0000 7ff89fb557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:50.968 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:50.967+0000 7ff89fb557c0 -1 bdev(0x558041c43c00 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:23:50.968 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:50.967+0000 7ff89fb557c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-10T20:23:56.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-10T20:23:56.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:23:56.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:23:56.930 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-10T20:23:56.930 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:23:57.543 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-10T20:23:57.543 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:23:57.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:23:57.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:23:57.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:23:57.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:23:57.575 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:57.575+0000 7f449e10a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:57.577 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:57.579+0000 7f449e10a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:57.580 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:57.579+0000 7f449e10a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:23:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:23:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:23:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:23:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:23:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:23:58.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:58.015 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:23:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:23:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:58.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:23:58.474 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:23:58.839 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:23:58.839+0000 7f449e10a7c0 -1 Falling back to public interface 2024-09-10T20:23:59.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:23:59.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:23:59.477 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:23:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:23:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:23:59.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:00.022 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:00.625 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:00.627+0000 7f449e10a7c0 -1 osd.3 0 log_to_monitors true 2024-09-10T20:24:01.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:01.023 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:01.023 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:24:01.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:24:01.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:01.024 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:01.468 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:02.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:02.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:02.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:24:02.470 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:24:02.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:02.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:02.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:03.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:24:03.990 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:24:03.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:03.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:04.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:05.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:05.560 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:05.560 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:24:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:24:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:05.561 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:06.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:07.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:24:07.015 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:24:07.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:07.016 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:07.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:08.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:08.556 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:08.557 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:24:08.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:24:08.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:08.557 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:09.002 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:10.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:10.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:10.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:24:10.005 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:24:10.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:10.005 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:10.535 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:11.537 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:11.537 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:24:11.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:11.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:24:11.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:11.538 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:12.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:13.027 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:13.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:13.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:24:13.028 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:24:13.028 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:13.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:13.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:14.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:24:14.511 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:24:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:14.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:15.383 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:16.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:16.385 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:16.385 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:24:16.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:24:16.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:16.386 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:17.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:18.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:18.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:18.315 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:24:18.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:24:18.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:18.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:19.359 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:20.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:20.361 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:24:20.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:24:20.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:20.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:20.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:21.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:21.864 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:24:21.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:24:21.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:21.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:22.828 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:23.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:23.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:23.830 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:24:23.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:24:23.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:23.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:24.330 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2883017414,v1:127.0.0.1:6827/2883017414] [v2:127.0.0.1:6828/2883017414,v1:127.0.0.1:6829/2883017414] exists,up 3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611 2024-09-10T20:24:24.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:24:24.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:24:24.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:24:24.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-10T20:24:24.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-10T20:24:25.806 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'ec' already exists 2024-09-10T20:24:25.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-10T20:24:27.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-10T20:24:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 88281 2024-09-10T20:24:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=90945 2024-09-10T20:24:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-10T20:24:27.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-10T20:24:28.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 90966' 2024-09-10T20:24:28.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-10T20:24:28.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 90967' 2024-09-10T20:24:28.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-10T20:24:28.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 90968' 2024-09-10T20:24:28.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-10T20:24:28.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-10T20:24:29.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-10T20:24:29.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:24:29.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:24:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:24:29.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:24:29.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:24:29.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:24:29.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 90945 90966 90967 90968 2024-09-10T20:24:29.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-10T20:24:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-10T20:24:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-10T20:24:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:24:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:24:29.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:29.287 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:24:29.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:24:29.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:24:29.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:24:29.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:24:29.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:24:29.289 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-10T20:24:29.292 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:24:29.293 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:24:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:24:29.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-10T20:24:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:24:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:24:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:24:29.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:24:29.315 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:29.314+0000 7fdaef0237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:29.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:29.318+0000 7fdaef0237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:29.320 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:29.318+0000 7fdaef0237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:29.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:24:29.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:24:29.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:24:29.743 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:24:29.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:24:29.744 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:24:29.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:29.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:24:29.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:29.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:24:30.233 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2355015296,v1:127.0.0.1:6803/2355015296] [v2:127.0.0.1:6804/2355015296,v1:127.0.0.1:6805/2355015296] exists,up d1d3b79b-e313-4d4a-b884-878c9ccb1d17 2024-09-10T20:24:30.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:24:30.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:24:30.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:24:30.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-10T20:24:30.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-10T20:24:30.236 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:24:30.237 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:24:30.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:24:30.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:24:30.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:24:30.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-10T20:24:30.239 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:24:30.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:24:30.267 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:30.266+0000 7fb9d45937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:30.270 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:30.270+0000 7fb9d45937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:30.271 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:30.270+0000 7fb9d45937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:30.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:30.326+0000 7fdaef0237c0 -1 Falling back to public interface 2024-09-10T20:24:30.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:24:30.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:24:30.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:24:30.762 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:24:30.762 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:24:30.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:24:30.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:30.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:24:30.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:30.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:24:31.454 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/683245533,v1:127.0.0.1:6811/683245533] [v2:127.0.0.1:6812/683245533,v1:127.0.0.1:6813/683245533] exists,up 3af975c8-2ba8-40ec-a8ed-fbdb0e18dc82 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:24:31.457 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:31.458 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:24:31.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-10T20:24:31.460 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:24:31.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:24:31.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:24:31.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-10T20:24:31.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:24:31.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:24:31.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:24:31.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:24:31.483 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:31.482+0000 7fa0cea4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:31.506 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:31.506+0000 7fa0cea4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:31.508 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:31.506+0000 7fa0cea4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:31.530 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:31.530+0000 7fb9d45937c0 -1 Falling back to public interface 2024-09-10T20:24:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:24:31.965 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:24:31.965 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:31.966 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:24:32.425 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/672733083,v1:127.0.0.1:6819/672733083] [v2:127.0.0.1:6820/672733083,v1:127.0.0.1:6821/672733083] exists,up d5f25bd3-76b8-4ffc-b421-f17d94966302 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ec-error-rollforward 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:24:32.429 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:32.430 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:24:32.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-10T20:24:32.432 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-10T20:24:32.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:24:32.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=07bf963d-6f1d-40eb-8f3b-0d5e12ae2c8b --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83789/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:24:32.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-10T20:24:32.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:24:32.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:24:32.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:24:32.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:24:32.454 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:32.454+0000 7ff9fabc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:32.532 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:32.530+0000 7ff9fabc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:32.534 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:32.534+0000 7ff9fabc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:32.790 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:32.790+0000 7fdaef0237c0 -1 osd.0 25 log_to_monitors true 2024-09-10T20:24:32.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:24:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:24:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:24:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:24:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:24:32.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:32.976 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:24:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:24:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:32.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:24:33.054 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:33.054+0000 7fa0cea4d7c0 -1 Falling back to public interface 2024-09-10T20:24:33.570 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2883017414,v1:127.0.0.1:6827/2883017414] [v2:127.0.0.1:6828/2883017414,v1:127.0.0.1:6829/2883017414] exists,up 3dbdc357-a9f7-42ac-b57f-c9cbcbaf7611 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:24:33.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:24:33.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:24:33.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:24:33.572 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:24:33.572 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:24:33.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:24:33.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:24:33.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:24:33.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:24:33.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:24:33.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:24:33.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:24:33.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:24:34.079 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:34.078+0000 7ff9fabc47c0 -1 Falling back to public interface 2024-09-10T20:24:34.156 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:24:34.156 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:24:34.157 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-10T20:24:34.157 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-10T20:24:34.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:24:34.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:24:34.157 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:24:34.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=120259084291 2024-09-10T20:24:34.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 120259084291 2024-09-10T20:24:34.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-120259084291' 2024-09-10T20:24:34.345 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:24:34.345 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:24:34.423 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:34.422+0000 7fb9d45937c0 -1 osd.1 25 log_to_monitors true 2024-09-10T20:24:36.117 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:36.118+0000 7fa0cea4d7c0 -1 osd.2 24 log_to_monitors true 2024-09-10T20:24:37.097 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:37.098+0000 7ff9fabc47c0 -1 osd.3 25 log_to_monitors true 2024-09-10T20:24:39.507 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=137438953475 2024-09-10T20:24:39.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 137438953475 2024-09-10T20:24:39.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-120259084291 1-137438953475' 2024-09-10T20:24:39.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:24:39.508 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:24:39.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=158913789955 2024-09-10T20:24:39.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 158913789955 2024-09-10T20:24:39.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-120259084291 1-137438953475 2-158913789955' 2024-09-10T20:24:39.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:24:39.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:24:39.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=171798691843 2024-09-10T20:24:39.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 171798691843 2024-09-10T20:24:39.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-120259084291 1-137438953475 2-158913789955 3-171798691843' 2024-09-10T20:24:39.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:24:39.846 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-120259084291 2024-09-10T20:24:39.847 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:24:39.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:24:39.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-120259084291 2024-09-10T20:24:39.848 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:24:39.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=120259084291 2024-09-10T20:24:39.849 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 120259084291 2024-09-10T20:24:39.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2024-09-10T20:24:39.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:24:40.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 120259084292 -lt 120259084291 2024-09-10T20:24:40.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:24:40.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-137438953475 2024-09-10T20:24:40.322 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:24:40.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:24:40.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-137438953475 2024-09-10T20:24:40.324 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:24:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=137438953475 2024-09-10T20:24:40.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-09-10T20:24:40.325 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 137438953475 2024-09-10T20:24:40.326 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:24:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 137438953475 2024-09-10T20:24:40.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:24:41.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:24:41.827 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:24:42.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 137438953476 -lt 137438953475 2024-09-10T20:24:42.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:24:42.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:24:42.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-158913789955 2024-09-10T20:24:42.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:24:42.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-158913789955 2024-09-10T20:24:42.313 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:24:42.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=158913789955 2024-09-10T20:24:42.314 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 158913789955 2024-09-10T20:24:42.315 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-09-10T20:24:42.315 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:24:42.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 158913789955 -lt 158913789955 2024-09-10T20:24:42.802 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:24:42.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-171798691843 2024-09-10T20:24:42.803 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:24:42.805 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:24:42.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-171798691843 2024-09-10T20:24:42.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:24:42.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=171798691843 2024-09-10T20:24:42.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 171798691843' 2024-09-10T20:24:42.808 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 171798691843 2024-09-10T20:24:42.808 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:24:43.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 171798691843 -lt 171798691843 2024-09-10T20:24:43.247 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:24:43.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:24:43.248 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:24:43.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:24:43.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:24:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:24:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:24:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:24:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:24:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:24:43.830 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:24:44.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:24:44.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:24:44.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:24:44.240 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:24:44.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-10T20:24:44.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:24:44.823 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:24:44.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:24:44.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:24:44.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:24:45.133 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:24:45.134 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:24:45.134 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:24:45.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:24:45.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:24:45.136 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:24:45.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:24:45.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:24:45.138 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:24:45.139 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:24:45.139 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:24:45.141 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:24:45.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:24:45.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-10T20:24:45.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:24:45.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:45.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:24:45.167 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.83789 2024-09-10T20:24:45.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:24:45.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:24:45.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:24:45.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/ec-error-rollforward 0 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ec-error-rollforward 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:24:45.169 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:24:45.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:24:45.172 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:24:45.172 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:24:45.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:24:45.173 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:24:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:24:45.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:24:45.175 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:24:45.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:24:45.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:24:45.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:24:45.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:24:45.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:24:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:24:45.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ec-error-rollforward 2024-09-10T20:24:45.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:24:45.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:45.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.83789 2024-09-10T20:24:45.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.83789 2024-09-10T20:24:45.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:24:45.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:24:45.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:24:45.183 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:24:45.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:24:45.241 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-10T20:24:45.241 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-10T20:24:45.297 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:24:45.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bench 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-10T20:24:45.303 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-10T20:24:45.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=74f1a15a-961b-4e9f-bd44-7a8e5b1df89b --auth-supported=none ' 2024-09-10T20:24:45.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-10T20:24:45.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-10T20:24:45.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-10T20:24:45.304 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:24:45.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bench 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bench 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:24:45.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:24:45.308 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:24:45.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:24:45.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:24:45.310 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:24:45.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:24:45.311 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:24:45.311 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:24:45.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:24:45.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:24:45.312 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:24:45.312 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:24:45.313 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:24:45.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:24:45.314 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-10T20:24:45.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:24:45.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:45.316 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:45.316 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.93473 2024-09-10T20:24:45.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:24:45.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:24:45.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bench 2024-09-10T20:24:45.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:24:45.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:45.318 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:45.318 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.93473 2024-09-10T20:24:45.320 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-10T20:24:45.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-10T20:24:45.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bench 2024-09-10T20:24:45.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:24:45.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:24:45.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:24:45.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bench/a 2024-09-10T20:24:45.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.93473/$cluster-$name.asok' 2024-09-10T20:24:45.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93473/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:24:46.014 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:24:46.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:24:46.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:24:46.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:24:46.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:24:46.016 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.93473/ceph-mon.a.asok 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:24:46.017 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.93473/ceph-mon.a.asok config get fsid 2024-09-10T20:24:46.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:24:46.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:24:46.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:24:46.135 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.93473/ceph-mon.a.asok 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:24:46.137 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.93473/ceph-mon.a.asok config get mon_host 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bench 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bench/x 2024-09-10T20:24:46.248 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:24:47.159 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:24:47.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:47.160 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:47.160 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:47.161 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:47.161 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:47.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.93473/$cluster-$name.asok' 2024-09-10T20:24:47.161 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:24:47.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93473/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bench 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bench/0 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=74f1a15a-961b-4e9f-bd44-7a8e5b1df89b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:24:47.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.93473/$cluster-$name.asok' 2024-09-10T20:24:47.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93473/$cluster-$name.asok' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:24:47.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bench/0 2024-09-10T20:24:47.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:24:47.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8cfb9263-3629-4799-9f31-6e88dac600f5 2024-09-10T20:24:47.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8cfb9263-3629-4799-9f31-6e88dac600f5' 2024-09-10T20:24:47.225 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 8cfb9263-3629-4799-9f31-6e88dac600f5 2024-09-10T20:24:47.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:24:47.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAPq+BmMnd3DhAAe7qZUFhf8jxp9Ta3RBsUeg== 2024-09-10T20:24:47.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAPq+BmMnd3DhAAe7qZUFhf8jxp9Ta3RBsUeg=="}' 2024-09-10T20:24:47.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8cfb9263-3629-4799-9f31-6e88dac600f5 -i td/osd-bench/0/new.json 2024-09-10T20:24:47.528 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:24:47.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bench/0/new.json 2024-09-10T20:24:47.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=74f1a15a-961b-4e9f-bd44-7a8e5b1df89b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPq+BmMnd3DhAAe7qZUFhf8jxp9Ta3RBsUeg== --osd-uuid 8cfb9263-3629-4799-9f31-6e88dac600f5 2024-09-10T20:24:47.579 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:47.578+0000 7f2b81b2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:47.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:47.586+0000 7f2b81b2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:47.588 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:47.586+0000 7f2b81b2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:47.589 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:47.590+0000 7f2b81b2b7c0 -1 bdev(0x55ae2f715c00 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:24:47.589 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:47.590+0000 7f2b81b2b7c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-10T20:24:53.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-10T20:24:53.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:24:53.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:24:53.380 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:24:53.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:24:54.003 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:24:54.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:24:54.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=74f1a15a-961b-4e9f-bd44-7a8e5b1df89b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93473/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:24:54.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:24:54.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:24:54.004 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:24:54.034 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:54.034+0000 7f21db44e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:54.037 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:54.034+0000 7f21db44e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:54.038 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:54.038+0000 7f21db44e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:54.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:24:54.432 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:24:54.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:54.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:24:54.905 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:55.288 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:55.286+0000 7f21db44e7c0 -1 Falling back to public interface 2024-09-10T20:24:55.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:55.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:55.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:24:55.907 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:24:55.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:55.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:24:56.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:56.883 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:24:56.882+0000 7f21db44e7c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:24:57.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:57.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:57.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:24:57.380 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:24:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:57.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:24:57.896 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:24:58.898 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:24:58.899 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:24:58.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:24:58.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:24:58.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:24:58.899 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:24:59.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:00.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:00.356 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:00.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:25:00.357 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:25:00.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:00.357 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:00.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:01.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:01.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:01.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:25:01.808 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:25:01.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:01.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:02.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:03.260 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:03.260 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:25:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:25:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:03.261 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:03.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:04.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:04.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:04.725 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:25:04.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:25:04.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:04.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:05.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:06.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:06.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:06.166 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:25:06.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:25:06.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:06.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:06.665 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:07.667 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:07.667 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:25:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:25:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:07.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:08.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:09.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:09.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:09.117 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:25:09.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:25:09.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:09.118 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:09.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:10.879 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:10.880 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:25:10.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:10.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:25:10.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:10.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:11.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:12.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:12.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:12.657 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:25:12.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:25:12.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:12.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:13.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:14.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:14.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:14.299 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:25:14.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:25:14.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:14.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:15.130 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:16.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:16.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:16.131 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:25:16.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:25:16.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:16.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:17.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:18.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:18.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:18.117 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:25:18.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:25:18.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:18.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:19.066 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:25:20.068 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:25:20.068 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:25:20.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:25:20.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:25:20.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:25:20.069 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:25:20.528 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3518804055,v1:127.0.0.1:6803/3518804055] [v2:127.0.0.1:6804/3518804055,v1:127.0.0.1:6805/3518804055] exists,up 8cfb9263-3629-4799-9f31-6e88dac600f5 2024-09-10T20:25:20.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:25:20.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:25:20.529 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:25:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-10T20:25:20.529 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-10T20:25:20.547 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-10T20:25:20.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-10T20:25:20.547 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-10T20:25:20.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-10T20:25:20.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-10T20:25:20.566 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-10T20:25:20.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-10T20:25:20.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-10T20:25:20.584 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-10T20:25:20.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-10T20:25:20.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-10T20:25:20.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-10T20:25:20.599 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T20:25:20.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_max_block_size 2024-09-10T20:25:20.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T20:25:20.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T20:25:20.600 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-10T20:25:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T20:25:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T20:25:20.738 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-10T20:25:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T20:25:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-10T20:25:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-10T20:25:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-10T20:25:20.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-10T20:25:20.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T20:25:20.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=bench_small_size_max_iops 2024-09-10T20:25:20.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T20:25:20.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T20:25:20.740 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-10T20:25:20.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T20:25:20.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T20:25:20.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-10T20:25:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T20:25:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-10T20:25:20.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: expect_failure: local dir=td/osd-bench 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: expect_failure: shift 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: expect_failure: shift 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: expect_failure: local success 2024-09-10T20:25:20.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-10T20:25:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: success=false 2024-09-10T20:25:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: false 2024-09-10T20:25:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-10T20:25:20.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: expect_failure: return 0 2024-09-10T20:25:20.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-10T20:26:10.187 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:26:10.188 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_written": 1073741824, 2024-09-10T20:26:10.188 INFO:tasks.workunit.client.0.smithi039.stdout: "blocksize": 4194304, 2024-09-10T20:26:10.188 INFO:tasks.workunit.client.0.smithi039.stdout: "elapsed_sec": 23.86810213, 2024-09-10T20:26:10.188 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_per_sec": 44986476.853155643, 2024-09-10T20:26:10.188 INFO:tasks.workunit.client.0.smithi039.stdout: "iops": 10.725611890114699 2024-09-10T20:26:10.188 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:26:10.214 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_written": 10, 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout: "blocksize": 14456, 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout: "elapsed_sec": 0.025116111, 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_per_sec": 398.15081244066806, 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout: "iops": 0.027542253212553131 2024-09-10T20:26:10.630 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:26:10.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout:{ 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_written": 1, 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout: "blocksize": 14456, 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout: "elapsed_sec": 0.022876456, 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout: "bytes_per_sec": 43.713064646027341, 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout: "iops": 0.0030238699948829098 2024-09-10T20:26:10.914 INFO:tasks.workunit.client.0.smithi039.stdout:} 2024-09-10T20:26:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-10T20:26:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-10T20:26:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:26:10.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-10T20:26:10.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:26:10.937 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:26:10.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:26:10.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:26:10.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:26:11.045 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:26:11.045 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:26:11.046 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:26:11.046 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:26:11.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:26:11.048 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:26:11.048 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:26:11.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:26:11.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:26:11.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:26:11.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:26:11.051 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:26:11.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:26:11.052 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-10T20:26:11.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:26:11.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:11.082 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:26:11.083 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.93473 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bench 0 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bench 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bench KILL 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:26:11.085 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:26:11.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:26:11.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:26:11.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:26:11.086 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:26:11.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:26:11.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:26:11.087 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:26:11.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:26:11.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:26:11.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:26:11.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:26:11.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:26:11.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:26:11.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:26:11.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:26:11.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:26:11.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bench 2024-09-10T20:26:11.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:26:11.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:11.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.93473 2024-09-10T20:26:11.093 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.93473 2024-09-10T20:26:11.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:26:11.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:26:11.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:26:11.095 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:26:11.096 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:26:11.152 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-10T20:26:11.153 DEBUG:teuthology.orchestra.run.smithi039:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-10T20:26:11.210 INFO:tasks.workunit.client.0.smithi039.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:26:11.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:26:11.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:26:11.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:26:11.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-bluefs-volume-ops 2024-09-10T20:26:11.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:11.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-10T20:26:11.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-10T20:26:11.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:26:11.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-10T20:26:11.221 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_bluestore2 2024-09-10T20:26:11.221 INFO:tasks.workunit.client.0.smithi039.stderr:TEST_bluestore_expand' 2024-09-10T20:26:11.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-10T20:26:11.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:26:11.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:26:11.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:26:11.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:26:11.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:26:11.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:26:11.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:26:11.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:26:11.226 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:26:11.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:26:11.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:26:11.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:26:11.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:26:11.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:26:11.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:26:11.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-10T20:26:11.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:26:11.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:11.232 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:11.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.95628 2024-09-10T20:26:11.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:26:11.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:26:11.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-10T20:26:11.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:26:11.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:11.235 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:11.235 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.95628 2024-09-10T20:26:11.237 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:26:11.238 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-10T20:26:11.239 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-10T20:26:11.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:11.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:26:11.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:26:11.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:26:11.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-10T20:26:11.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:11.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:26:11.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:26:11.725 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:26:11.725 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:26:11.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:11.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:11.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:26:11.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:26:11.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:26:11.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:26:11.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:26:11.929 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:26:11.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:26:11.930 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:26:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:26:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:26:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:26:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:26:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:11.932 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:11.933 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.95628/ceph-mon.a.asok 2024-09-10T20:26:11.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:26:11.933 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.95628/ceph-mon.a.asok config get fsid 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:26:12.050 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:26:12.051 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:26:12.051 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:26:12.051 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:12.051 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:12.052 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.95628/ceph-mon.a.asok 2024-09-10T20:26:12.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:26:12.052 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.95628/ceph-mon.a.asok config get mon_host 2024-09-10T20:26:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-10T20:26:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:26:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:26:12.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:26:12.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-10T20:26:12.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:26:13.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:26:13.079 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:26:13.080 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:26:13.080 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:26:13.080 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:13.080 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:13.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:26:13.081 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:26:13.082 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:26:13.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:26:13.143 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:26:13.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:26:13.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:26:13.146 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:26:13.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:26:13.147 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:26:13.148 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0618f729-b530-4370-b507-bc57485ba1dd' 2024-09-10T20:26:13.148 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:26:13.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBlq+BmMojACRAAh16qeQ9X89EpwdUGkap4pw== 2024-09-10T20:26:13.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBlq+BmMojACRAAh16qeQ9X89EpwdUGkap4pw=="}' 2024-09-10T20:26:13.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0618f729-b530-4370-b507-bc57485ba1dd -i td/osd-bluefs-volume-ops/0/new.json 2024-09-10T20:26:13.448 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:26:13.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-10T20:26:13.465 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlq+BmMojACRAAh16qeQ9X89EpwdUGkap4pw== --osd-uuid 0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:26:13.493 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:13.492+0000 7fd1f488b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:13.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:13.512+0000 7fd1f488b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:13.517 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:13.512+0000 7fd1f488b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:13.517 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:13.516+0000 7fd1f488b7c0 -1 bdev(0x557913321c00 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:26:13.518 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:13.516+0000 7fd1f488b7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-10T20:26:23.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-10T20:26:23.208 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:26:23.210 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:26:23.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:26:23.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:26:23.817 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:26:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:26:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:26:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:26:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:26:23.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:26:23.849 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:23.851+0000 7fd591ae37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:23.852 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:23.851+0000 7fd591ae37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:23.853 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:23.851+0000 7fd591ae37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:24.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:26:24.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:26:24.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:26:24.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:26:24.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:26:24.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:24.273 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:26:24.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:26:24.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:24.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:24.757 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:25.597 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:25.595+0000 7fd591ae37c0 -1 Falling back to public interface 2024-09-10T20:26:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:25.758 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:26:25.759 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:26:25.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:25.759 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:26.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:27.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:27.242 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:26:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:26:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:27.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:27.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:28.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:28.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:28.725 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:26:28.725 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:26:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:28.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:29.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:29.668 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:29.667+0000 7fd591ae37c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:26:30.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:30.173 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:30.173 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:26:30.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:26:30.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:30.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:30.668 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:31.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:26:31.671 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:26:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:31.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:32.115 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:33.116 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:33.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:33.121 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:26:33.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:26:33.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:33.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:33.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:34.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:34.575 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:26:34.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:26:34.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:34.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:35.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:36.041 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:26:36.042 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:26:36.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:36.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:36.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:37.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:37.558 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:37.558 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:26:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:26:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:37.559 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:38.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:39.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:39.019 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:39.020 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:26:39.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:26:39.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:39.020 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:39.473 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:40.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:40.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:40.475 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:26:40.475 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:26:40.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:40.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:40.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:41.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:41.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:41.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:26:41.982 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:26:41.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:41.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:42.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:43.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:43.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:43.670 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:26:43.671 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:26:43.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:43.671 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:44.655 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:45.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:45.656 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:45.657 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:26:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:26:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:45.657 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:46.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:47.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:47.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:47.492 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:26:47.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:26:47.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:47.493 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:48.488 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:49.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:49.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:49.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:26:49.489 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:26:49.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:49.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:50.256 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:51.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:51.258 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:51.258 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:26:51.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:26:51.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:51.259 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:52.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:26:53.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:26:53.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:26:53.039 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-10T20:26:53.039 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:26:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:26:53.040 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:26:53.480 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/767399868,v1:127.0.0.1:6803/767399868] [v2:127.0.0.1:6804/767399868,v1:127.0.0.1:6805/767399868] exists,up 0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:26:53.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:26:53.480 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:26:53.481 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:26:53.481 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:26:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=97094 2024-09-10T20:26:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-10T20:26:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:26:53.484 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:26:53.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:26:53.486 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:26:53.487 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-10T20:26:53.509 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:26:53.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:26:53.510 INFO:tasks.workunit.client.0.smithi039.stdout:add osd1 5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:26:53.510 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 5569bcd7-ae19-4a16-91c5-cb40384eedda' 2024-09-10T20:26:53.511 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:26:53.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCNq+Bmk8bDHxAAsXeaLu8zWGRc0S0lvwW8CQ== 2024-09-10T20:26:53.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCNq+Bmk8bDHxAAsXeaLu8zWGRc0S0lvwW8CQ=="}' 2024-09-10T20:26:53.532 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5569bcd7-ae19-4a16-91c5-cb40384eedda -i td/osd-bluefs-volume-ops/1/new.json 2024-09-10T20:26:54.098 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:26:54.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-10T20:26:54.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNq+Bmk8bDHxAAsXeaLu8zWGRc0S0lvwW8CQ== --osd-uuid 5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:26:54.155 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:54.155+0000 7fe7241b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:54.179 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:54.179+0000 7fe7241b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:54.179+0000 7fe7241b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:26:54.180 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:54.179+0000 7fe7241b57c0 -1 bdev(0x55566c1cdc00 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:26:54.181 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:26:54.179+0000 7fe7241b57c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-10T20:27:03.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-10T20:27:03.647 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:27:03.648 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd1 key to auth repository 2024-09-10T20:27:03.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:27:03.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:27:04.241 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:27:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:27:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:27:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:27:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:27:04.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:27:04.273 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:04.270+0000 7f688c52d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:04.277 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:04.274+0000 7f688c52d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:04.278 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:04.278+0000 7f688c52d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:04.706 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:27:04.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:27:04.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:04.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:05.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:06.036 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:06.034+0000 7f688c52d7c0 -1 Falling back to public interface 2024-09-10T20:27:06.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:06.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:06.204 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:27:06.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:27:06.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:06.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:06.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:07.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:07.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:07.696 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:27:07.696 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:27:07.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:07.697 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:08.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:09.220 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:09.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:09.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:27:09.221 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:27:09.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:09.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:09.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:10.249 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:10.234+0000 7f688c52d7c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:27:10.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:10.711 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:10.711 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:27:10.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:27:10.712 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:10.713 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:11.404 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:12.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:12.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:12.406 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:27:12.406 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:27:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:12.407 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:12.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:13.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:13.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:13.868 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:27:13.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:27:13.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:13.869 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:14.563 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:15.564 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:15.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:15.565 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:27:15.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:27:15.565 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:15.566 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:16.158 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:17.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:17.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:17.160 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:27:17.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:27:17.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:17.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:17.611 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:18.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:18.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:18.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:27:18.614 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:27:18.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:18.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:19.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:20.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:20.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:20.100 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:27:20.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:27:20.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:20.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:20.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:21.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:27:21.692 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:27:21.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:21.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:22.240 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:23.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:23.241 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:23.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:27:23.242 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:27:23.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:23.242 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:23.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:24.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:24.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:24.924 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:27:24.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:27:24.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:24.925 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:25.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:26.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:26.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:26.516 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:27:26.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:27:26.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:26.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:27.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:28.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:28.848 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:27:28.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:27:28.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:28.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:29.739 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:30.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:30.741 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:30.742 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:27:30.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:27:30.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:30.742 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:31.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:32.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:32.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:32.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:27:32.724 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:27:32.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:32.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:33.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:34.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:34.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:34.233 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-10T20:27:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:27:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:34.234 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:27:34.719 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3798755208,v1:127.0.0.1:6811/3798755208] [v2:127.0.0.1:6812/3798755208,v1:127.0.0.1:6813/3798755208] exists,up 5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:27:34.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:27:34.719 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:27:34.720 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:27:34.720 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-10T20:27:34.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=99615 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:27:34.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:27:34.724 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:27:34.725 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:27:34.726 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:27:34.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:27:34.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:27:34.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-10T20:27:34.727 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:27:34.727 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:27:34.727 INFO:tasks.workunit.client.0.smithi039.stdout:add osd2 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:27:34.728 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 773ff91d-32bb-49f0-b7cc-3a7164b0ec86' 2024-09-10T20:27:34.728 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:27:34.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2q+BmdtxlLBAAdkyWWUDL3faJ6PeFCJk1fw== 2024-09-10T20:27:34.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2q+BmdtxlLBAAdkyWWUDL3faJ6PeFCJk1fw=="}' 2024-09-10T20:27:34.744 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-10T20:27:35.251 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:27:35.268 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-10T20:27:35.269 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2q+BmdtxlLBAAdkyWWUDL3faJ6PeFCJk1fw== --osd-uuid 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:27:35.305 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:35.301+0000 7f9fabf4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:35.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:35.305+0000 7f9fabf4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:35.308 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:35.305+0000 7f9fabf4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:35.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:35.305+0000 7f9fabf4c7c0 -1 bdev(0x55d162043c00 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:27:35.309 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:35.305+0000 7f9fabf4c7c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-10T20:27:45.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-10T20:27:45.195 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:27:45.196 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd2 key to auth repository 2024-09-10T20:27:45.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:27:45.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:27:45.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:27:45.788 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:27:45.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:27:45.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:27:45.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:27:45.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:27:45.821 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:45.817+0000 7f95abe737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:45.840 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:45.837+0000 7f95abe737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:45.841 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:45.837+0000 7f95abe737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:27:46.253 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:27:46.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:27:46.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:27:46.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:27:46.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:27:46.254 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:27:46.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:46.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:27:46.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:46.255 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:46.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:47.637 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:47.633+0000 7f95abe737c0 -1 Falling back to public interface 2024-09-10T20:27:47.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:47.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:47.722 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:27:47.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:27:47.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:47.723 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:48.159 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:49.160 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:49.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:49.161 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:27:49.161 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:27:49.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:49.162 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:49.649 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:50.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:50.650 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:27:50.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:50.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:27:50.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:50.651 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:51.098 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:51.643 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:27:51.641+0000 7f95abe737c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:27:52.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:52.100 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:52.100 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:27:52.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:27:52.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:52.101 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:52.745 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:53.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:53.748 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:27:53.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:27:53.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:53.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:54.196 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:55.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:55.198 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:55.198 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:27:55.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:27:55.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:55.199 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:55.797 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:56.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:56.799 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:56.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:27:56.800 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:27:56.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:56.800 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:57.288 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:58.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:58.290 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:58.291 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:27:58.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:27:58.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:58.291 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:27:58.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:27:59.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:27:59.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:27:59.891 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:27:59.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:27:59.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:27:59.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:00.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:01.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:01.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:01.366 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:28:01.366 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:28:01.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:01.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:01.818 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:02.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:02.820 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:02.820 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:28:02.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:28:02.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:02.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:03.271 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:04.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:04.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:04.273 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:28:04.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:28:04.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:04.274 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:04.929 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:05.931 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:05.931 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:28:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:28:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:05.932 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:06.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:07.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:07.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:07.687 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:28:07.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:28:07.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:07.688 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:08.514 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:09.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:09.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:09.516 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:28:09.516 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:28:09.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:09.517 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:10.244 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:11.246 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:11.246 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:28:11.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:11.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:28:11.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:11.247 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:11.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:12.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:12.978 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:28:12.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:12.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:28:12.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:12.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:13.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:14.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:14.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:14.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:28:14.889 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-10T20:28:14.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:14.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:28:15.373 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1738185456,v1:127.0.0.1:6819/1738185456] [v2:127.0.0.1:6820/1738185456,v1:127.0.0.1:6821/1738185456] exists,up 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:28:15.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:28:15.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:28:15.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:28:15.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-10T20:28:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=102138 2024-09-10T20:28:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-10T20:28:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:28:15.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:28:15.377 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:28:15.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:28:15.378 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:28:15.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-10T20:28:15.380 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:28:15.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:28:15.382 INFO:tasks.workunit.client.0.smithi039.stdout:add osd3 35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:28:15.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 35fcc31c-2a74-4f0c-8e4d-df88d69846db' 2024-09-10T20:28:15.382 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:28:15.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDfq+BmdL0IGBAAt6Mq81WU2Uy0yMY7lDnERQ== 2024-09-10T20:28:15.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDfq+BmdL0IGBAAt6Mq81WU2Uy0yMY7lDnERQ=="}' 2024-09-10T20:28:15.403 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 35fcc31c-2a74-4f0c-8e4d-df88d69846db -i td/osd-bluefs-volume-ops/3/new.json 2024-09-10T20:28:15.993 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:28:16.011 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-10T20:28:16.013 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDfq+BmdL0IGBAAt6Mq81WU2Uy0yMY7lDnERQ== --osd-uuid 35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:28:16.048 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:16.044+0000 7f1270c957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:16.050 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:16.044+0000 7f1270c957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:16.051 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:16.048+0000 7f1270c957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:16.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:16.048+0000 7f1270c957c0 -1 bdev(0x56353c129c00 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:28:16.052 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:16.048+0000 7f1270c957c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-10T20:28:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-10T20:28:26.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:28:26.193 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd3 key to auth repository 2024-09-10T20:28:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:28:26.193 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:28:26.782 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-10T20:28:26.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:28:26.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:28:26.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:28:26.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:28:26.783 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:28:26.813 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:26.808+0000 7f26fe1417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:26.823 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:26.820+0000 7f26fe1417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:26.824 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:26.820+0000 7f26fe1417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:28:27.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:28:27.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:28:27.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:28:27.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:28:27.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:28:27.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:27.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:28:27.263 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:28:27.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:27.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:27.763 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:28.642 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:28.636+0000 7f26fe1417c0 -1 Falling back to public interface 2024-09-10T20:28:28.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:28.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:28.765 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:28:28.766 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:28:28.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:28.766 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:29.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:30.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:30.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:30.204 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:28:30.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:28:30.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:30.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:30.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:31.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:31.692 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:31.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:28:31.693 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:28:31.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:31.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:32.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:32.697 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:28:32.692+0000 7f26fe1417c0 -1 osd.3 0 log_to_monitors true 2024-09-10T20:28:33.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:33.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:33.146 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:28:33.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:28:33.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:33.146 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:33.839 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:34.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:34.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:34.841 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:28:34.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:28:34.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:34.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:35.317 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:36.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:36.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:36.320 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:28:36.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:28:36.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:36.321 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:36.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:37.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:37.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:37.888 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:28:37.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:28:37.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:37.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:38.351 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:39.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:39.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:39.353 INFO:tasks.workunit.client.0.smithi039.stdout:8 2024-09-10T20:28:39.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:28:39.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:39.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:39.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:40.842 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:40.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:40.843 INFO:tasks.workunit.client.0.smithi039.stdout:9 2024-09-10T20:28:40.843 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:28:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:40.844 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:41.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:42.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:42.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:42.298 INFO:tasks.workunit.client.0.smithi039.stdout:10 2024-09-10T20:28:42.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:28:42.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:42.299 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:42.878 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:43.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:43.880 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:43.880 INFO:tasks.workunit.client.0.smithi039.stdout:11 2024-09-10T20:28:43.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:28:43.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:43.881 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:44.361 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:45.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:45.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:45.363 INFO:tasks.workunit.client.0.smithi039.stdout:12 2024-09-10T20:28:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:28:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:45.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:46.263 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:47.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:47.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:47.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:28:47.265 INFO:tasks.workunit.client.0.smithi039.stdout:13 2024-09-10T20:28:47.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:47.266 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:48.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:49.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:49.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:49.393 INFO:tasks.workunit.client.0.smithi039.stdout:14 2024-09-10T20:28:49.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:28:49.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:49.394 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:49.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:50.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:50.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:50.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:28:50.996 INFO:tasks.workunit.client.0.smithi039.stdout:15 2024-09-10T20:28:50.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:50.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:51.975 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:52.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:52.977 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:52.978 INFO:tasks.workunit.client.0.smithi039.stdout:16 2024-09-10T20:28:52.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:28:52.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:52.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:53.703 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:54.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:54.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:54.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:28:54.705 INFO:tasks.workunit.client.0.smithi039.stdout:17 2024-09-10T20:28:54.705 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:54.706 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:55.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:28:56.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:28:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:28:56.635 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 18 2024-09-10T20:28:56.635 INFO:tasks.workunit.client.0.smithi039.stdout:18 2024-09-10T20:28:56.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:28:56.636 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:28:57.075 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2061173048,v1:127.0.0.1:6827/2061173048] [v2:127.0.0.1:6828/2061173048,v1:127.0.0.1:6829/2061173048] exists,up 35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:28:57.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:28:57.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:28:57.076 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:28:57.077 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-10T20:28:57.077 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=104659 2024-09-10T20:28:57.078 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-10T20:29:02.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-10T20:29:02.079 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-10T20:29:02.955 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-10T20:29:02.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:29:03.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-10T20:29:04.021 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-10T20:29:04.021 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-10T20:29:04.021 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_105991 2024-09-10T20:29:04.021 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:29:04.021 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:29:05.021 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 64 48 0.187488 0.1875 0.434353 0.279502 2024-09-10T20:29:06.021 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 122 106 0.207008 0.226562 0.250205 0.284903 2024-09-10T20:29:07.021 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 185 169 0.220024 0.246094 0.211805 0.273236 2024-09-10T20:29:08.024 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 250 234 0.228484 0.253906 0.256613 0.26525 2024-09-10T20:29:09.022 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 308 292 0.228093 0.226562 0.218066 0.263069 2024-09-10T20:29:10.022 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 372 356 0.231739 0.25 0.22203 0.263457 2024-09-10T20:29:11.022 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 436 420 0.234344 0.25 0.253118 0.260162 2024-09-10T20:29:12.022 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 504 488 0.238249 0.265625 0.265813 0.258161 2024-09-10T20:29:13.022 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 568 552 0.23955 0.25 0.234037 0.257368 2024-09-10T20:29:14.022 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 639 623 0.243326 0.277344 0.215239 0.254806 2024-09-10T20:29:15.023 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 703 687 0.243929 0.25 0.255097 0.253056 2024-09-10T20:29:16.022 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 764 748 0.243455 0.238281 0.245241 0.254394 2024-09-10T20:29:17.025 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 825 809 0.243055 0.238281 0.23177 0.254898 2024-09-10T20:29:18.023 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 889 873 0.243549 0.25 0.262258 0.254233 2024-09-10T20:29:19.023 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 952 936 0.243717 0.246094 0.254534 0.255192 2024-09-10T20:29:20.023 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 1001 985 0.240446 0.191406 0.235329 0.258696 2024-09-10T20:29:21.023 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 1055 1039 0.238708 0.210938 0.25255 0.260364 2024-09-10T20:29:22.023 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 1115 1099 0.238466 0.234375 0.240967 0.260035 2024-09-10T20:29:23.023 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 1172 1156 0.237633 0.222656 0.288705 0.262003 2024-09-10T20:29:24.024 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:29:24.024372+0000 min lat: 0.172774 max lat: 0.522054 avg lat: 0.260963 2024-09-10T20:29:24.024 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:29:24.024 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 1237 1221 0.238445 0.253906 0.172774 0.260963 2024-09-10T20:29:25.024 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 1293 1277 0.237506 0.21875 0.249393 0.260918 2024-09-10T20:29:26.024 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 1365 1349 0.239493 0.28125 0.239504 0.26009 2024-09-10T20:29:27.024 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 1429 1413 0.239948 0.25 0.21677 0.259408 2024-09-10T20:29:28.024 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 1493 1477 0.240366 0.25 0.253366 0.259109 2024-09-10T20:29:29.024 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 1550 1534 0.239656 0.222656 0.252836 0.259033 2024-09-10T20:29:30.024 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 1601 1585 0.2381 0.199219 0.408905 0.260489 2024-09-10T20:29:31.025 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 1669 1653 0.239118 0.265625 0.252237 0.260606 2024-09-10T20:29:32.025 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 1727 1711 0.238669 0.226562 0.285538 0.260954 2024-09-10T20:29:33.025 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 1782 1766 0.237847 0.214844 0.24336 0.261765 2024-09-10T20:29:34.025 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 1841 1825 0.2376 0.230469 0.239972 0.261703 2024-09-10T20:29:35.025 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.2105 2024-09-10T20:29:35.025 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 1841 2024-09-10T20:29:35.025 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-10T20:29:35.025 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-10T20:29:35.025 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.238043 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0230318 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.28125 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.1875 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 60 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 5.89613 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 72 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 48 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.261608 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0416385 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.522054 2024-09-10T20:29:35.026 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.165894 2024-09-10T20:29:35.163 INFO:tasks.workunit.client.0.smithi039.stdout:after bench 2024-09-10T20:29:35.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-10T20:29:35.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 97094 2024-09-10T20:29:35.164 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-10T20:29:36.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 97094 2024-09-10T20:29:36.165 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (97094) - No such process 2024-09-10T20:29:36.166 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-10T20:29:36.846 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:29:36.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 99615 2024-09-10T20:29:36.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-10T20:29:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 99615 2024-09-10T20:29:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (99615) - No such process 2024-09-10T20:29:37.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-10T20:29:38.406 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:29:38.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 102138 2024-09-10T20:29:38.426 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-10T20:29:39.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 102138 2024-09-10T20:29:39.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (102138) - No such process 2024-09-10T20:29:39.428 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-10T20:29:39.894 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:29:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 104659 2024-09-10T20:29:39.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-10T20:29:40.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 104659 2024-09-10T20:29:40.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (104659) - No such process 2024-09-10T20:29:40.913 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-10T20:29:41.463 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 is already down. 2024-09-10T20:29:41.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:29:43.851 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:29:43.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:29:46.247 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:29:46.273 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:29:48.651 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:29:48.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:29:51.056 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:29:51.080 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-10T20:29:51.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-10T20:29:51.125 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:29:52.333 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-10T20:29:52.333 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-10T20:29:52.333 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100000000 : using 0x805b0000(2.0 GiB) 2024-09-10T20:29:52.333 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-10T20:29:52.333 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-10T20:29:52.337 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-10T20:29:52.337 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-10T20:29:52.346 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-10T20:29:52.346 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-10T20:29:52.487 INFO:tasks.workunit.client.0.smithi039.stdout:2 : size label updated to 4294967296 2024-09-10T20:29:58.145 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-10T20:29:58.147 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-10T20:29:58.177 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:29:59.378 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-10T20:29:59.378 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-10T20:29:59.378 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x101000000 : using 0x81590000(2.0 GiB) 2024-09-10T20:29:59.378 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-10T20:29:59.379 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-10T20:29:59.389 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-10T20:29:59.389 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-10T20:29:59.397 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-10T20:29:59.397 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-10T20:29:59.539 INFO:tasks.workunit.client.0.smithi039.stdout:2 : size label updated to 4311744512 2024-09-10T20:30:04.945 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-10T20:30:04.946 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-10T20:30:04.976 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:06.176 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-10T20:30:06.176 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-10T20:30:06.176 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100020000 : using 0x805d1000(2.0 GiB) 2024-09-10T20:30:06.176 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-10T20:30:06.176 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-10T20:30:06.197 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-10T20:30:06.197 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-10T20:30:06.205 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-10T20:30:06.205 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-10T20:30:06.447 INFO:tasks.workunit.client.0.smithi039.stdout:2 : size label updated to 4295098368 2024-09-10T20:30:11.886 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-10T20:30:11.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-10T20:30:11.915 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:13.124 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-09-10T20:30:13.125 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-10T20:30:13.125 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0xfff00000 : using 0x804b3000(2.0 GiB) 2024-09-10T20:30:13.125 INFO:tasks.workunit.client.0.smithi039.stdout:Expanding DB/WAL... 2024-09-10T20:30:13.125 INFO:tasks.workunit.client.0.smithi039.stdout:0 : expanding to 0x536870912 2024-09-10T20:30:13.149 INFO:tasks.workunit.client.0.smithi039.stdout:0 : size label updated to 536870912 2024-09-10T20:30:13.149 INFO:tasks.workunit.client.0.smithi039.stdout:1 : expanding to 0x1073741824 2024-09-10T20:30:13.154 INFO:tasks.workunit.client.0.smithi039.stdout:1 : size label updated to 1073741824 2024-09-10T20:30:13.154 INFO:tasks.workunit.client.0.smithi039.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-10T20:30:13.262 INFO:tasks.workunit.client.0.smithi039.stdout:2 : size label updated to 4293918720 2024-09-10T20:30:18.850 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:30:21.220 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:21.243 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:30:23.612 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:23.634 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:30:26.012 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:26.035 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:30:28.404 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:28.427 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-10T20:30:28.456 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:30.542 INFO:tasks.workunit.client.0.smithi039.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-09-10T20:30:30.542 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-09-10T20:30:30.542 INFO:tasks.workunit.client.0.smithi039.stdout:2 : device size 0x100000000 : using 0x5b0000(5.7 MiB) 2024-09-10T20:30:30.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-10T20:30:30.573 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:33.484 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-10T20:30:33.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:30:35.368 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:35.388 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-10T20:30:35.416 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:38.356 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-10T20:30:38.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:30:40.232 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:40.254 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-10T20:30:40.281 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:43.212 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-10T20:30:43.212 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-10T20:30:43.222 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:30:44.592 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:44.606 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-10T20:30:44.634 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:45.488 INFO:tasks.workunit.client.0.smithi039.stderr:Migrate to WAL device isn't supported. 2024-09-10T20:30:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-10T20:30:45.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:30:47.864 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:47.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-10T20:30:47.913 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:50.844 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-10T20:30:50.854 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:30:52.756 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:52.777 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-10T20:30:52.807 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:30:54.934 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:30:56.808 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:30:56.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:30:56.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:30:56.829 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:30:56.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:30:56.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:30:56.831 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:30:56.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:30:56.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:30:56.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:30:56.832 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:30:56.832 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:30:56.833 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:30:56.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:30:56.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:30:56.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:30:56.834 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:30:56.834 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:30:56.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:30:56.835 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:30:56.835 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-10T20:30:56.836 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:30:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:30:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:30:56.837 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:30:56.869 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:30:56.864+0000 7faec3ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:56.872 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:30:56.868+0000 7faec3ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:56.874 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:30:56.868+0000 7faec3ec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:30:57.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:30:57.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:30:57.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:30:57.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:30:57.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:30:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:30:57.324 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:30:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:57.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:57.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:30:58.638 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:30:58.632+0000 7faec3ec47c0 -1 Falling back to public interface 2024-09-10T20:30:58.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:30:58.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:30:58.809 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:30:58.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:30:58.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:30:58.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:30:59.300 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:00.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:00.302 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:00.302 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:31:00.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:31:00.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:00.303 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:31:00.786 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:01.593 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:01.588+0000 7faec3ec47c0 -1 osd.0 27 log_to_monitors true 2024-09-10T20:31:01.788 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:01.788 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:31:01.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:01.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:31:01.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:01.789 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:31:02.271 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 38 up_thru 22 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/4250965817,v1:127.0.0.1:6803/4250965817] [v2:127.0.0.1:6804/4250965817,v1:127.0.0.1:6805/4250965817] exists,up 0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:31:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:31:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:31:02.272 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:31:02.273 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=118134 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:31:02.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:31:02.277 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:31:02.279 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:31:02.279 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:31:02.279 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:31:02.279 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:31:02.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:31:02.279 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:31:02.280 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:31:02.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:31:02.281 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-10T20:31:02.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:31:02.281 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:31:02.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:31:02.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:31:02.317 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:02.312+0000 7fa4f624e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:02.354 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:02.348+0000 7fa4f624e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:02.355 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:02.348+0000 7fa4f624e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:02.721 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:31:02.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:31:02.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:02.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:03.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:03.894 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:03.888+0000 7fa4f624e7c0 -1 Falling back to public interface 2024-09-10T20:31:04.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:04.206 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:04.206 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:31:04.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:31:04.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:04.207 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:04.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:05.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:05.754 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:05.754 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:31:05.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:31:05.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:05.755 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:06.262 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:06.752 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:06.748+0000 7fa4f624e7c0 -1 osd.1 30 log_to_monitors true 2024-09-10T20:31:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:07.264 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:31:07.264 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:31:07.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:07.265 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:31:07.987 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/3091970304,v1:127.0.0.1:6811/3091970304] [v2:127.0.0.1:6812/3091970304,v1:127.0.0.1:6813/3091970304] exists,up 5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:31:07.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:31:07.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:31:07.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:31:07.988 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-10T20:31:07.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=119074 2024-09-10T20:31:07.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-10T20:31:07.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:31:07.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:31:07.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:31:07.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:31:07.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:31:07.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:31:07.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:31:07.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:31:07.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:31:07.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:31:07.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:31:07.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-10T20:31:07.993 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:31:07.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:31:07.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:31:07.994 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-10T20:31:07.995 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:31:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:31:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:31:07.996 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:31:08.027 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:08.020+0000 7fa801ec27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:08.108 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:08.104+0000 7fa801ec27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:08.109 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:08.104+0000 7fa801ec27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:31:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:31:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:31:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:31:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:31:08.544 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:08.545 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:31:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:31:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:08.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:09.008 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:09.343 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:09.340+0000 7fa801ec27c0 -1 Falling back to public interface 2024-09-10T20:31:10.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:10.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:10.010 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:31:10.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:31:10.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:10.010 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:10.476 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:11.425 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:11.420+0000 7fa801ec27c0 -1 osd.2 33 log_to_monitors true 2024-09-10T20:31:11.477 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:11.478 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:31:11.478 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:11.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:31:11.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:31:11.479 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:12.116 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 45 up_thru 31 down_at 34 last_clean_interval [15,33) [v2:127.0.0.1:6818/746980028,v1:127.0.0.1:6819/746980028] [v2:127.0.0.1:6820/746980028,v1:127.0.0.1:6821/746980028] exists,up 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:31:12.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:31:12.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:31:12.117 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:31:12.117 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-10T20:31:12.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=120015 2024-09-10T20:31:12.119 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-10T20:31:12.120 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:31:12.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:31:12.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:31:12.122 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-10T20:31:12.123 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-10T20:31:12.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:31:12.124 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:31:12.124 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-10T20:31:12.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:31:12.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:31:12.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:31:12.126 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:31:12.163 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:12.159+0000 7ff60f8a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:12.279 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:12.275+0000 7ff60f8a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:12.281 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:12.279+0000 7ff60f8a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:31:12.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:31:12.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:31:12.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:31:12.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:31:12.821 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:31:12.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:12.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:31:12.822 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:31:12.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:12.822 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:31:13.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:13.825 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:13.819+0000 7ff60f8a17c0 -1 Falling back to public interface 2024-09-10T20:31:14.441 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:31:14.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:14.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:14.441 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:31:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:31:14.442 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:14.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:15.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:15.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:15.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:31:15.888 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:31:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:15.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:31:16.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:31:16.896 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:31:16.895+0000 7ff60f8a17c0 -1 osd.3 35 log_to_monitors true 2024-09-10T20:31:17.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:31:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:31:17.369 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:31:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:31:17.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:31:17.370 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:31:18.091 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 50 up_thru 50 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/4196222582,v1:127.0.0.1:6827/4196222582] [v2:127.0.0.1:6828/4196222582,v1:127.0.0.1:6829/4196222582] exists,up 35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:31:18.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:31:18.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:31:18.092 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:31:18.093 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-10T20:31:18.094 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=120942 2024-09-10T20:31:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-10T20:31:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:31:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:31:18.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:31:18.095 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:31:18.096 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:31:18.096 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:31:18.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:31:18.096 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:31:18.097 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:31:18.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:31:18.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:31:18.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:31:18.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:31:18.217 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:31:18.217 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:31:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:31:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:31:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-10T20:31:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-10T20:31:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:31:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:18.777 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:31:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=163208757254 2024-09-10T20:31:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 163208757254 2024-09-10T20:31:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757254' 2024-09-10T20:31:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:18.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:31:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=176093659141 2024-09-10T20:31:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 176093659141 2024-09-10T20:31:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757254 1-176093659141' 2024-09-10T20:31:19.144 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:19.145 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:31:19.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=193273528324 2024-09-10T20:31:19.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 193273528324 2024-09-10T20:31:19.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757254 1-176093659141 2-193273528324' 2024-09-10T20:31:19.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:31:19.297 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:31:19.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=214748364803 2024-09-10T20:31:19.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 214748364803 2024-09-10T20:31:19.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-163208757254 1-176093659141 2-193273528324 3-214748364803' 2024-09-10T20:31:19.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:19.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-163208757254 2024-09-10T20:31:19.451 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:19.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:31:19.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-163208757254 2024-09-10T20:31:19.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:19.454 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 163208757254 2024-09-10T20:31:19.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=163208757254 2024-09-10T20:31:19.454 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 163208757254' 2024-09-10T20:31:19.455 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:31:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757252 -lt 163208757254 2024-09-10T20:31:19.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:31:20.937 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:31:20.938 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:31:21.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 163208757254 -lt 163208757254 2024-09-10T20:31:21.447 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:21.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:21.448 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-176093659141 2024-09-10T20:31:21.449 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:31:21.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-176093659141 2024-09-10T20:31:21.450 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:21.451 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=176093659141 2024-09-10T20:31:21.452 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 176093659141 2024-09-10T20:31:21.452 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-09-10T20:31:21.452 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:31:21.919 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 176093659141 -lt 176093659141 2024-09-10T20:31:21.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:21.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:21.920 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-193273528324 2024-09-10T20:31:21.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:31:21.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-193273528324 2024-09-10T20:31:21.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:21.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=193273528324 2024-09-10T20:31:21.924 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 193273528324' 2024-09-10T20:31:21.924 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 193273528324 2024-09-10T20:31:21.925 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 193273528325 -lt 193273528324 2024-09-10T20:31:22.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:31:22.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-214748364803 2024-09-10T20:31:22.365 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:31:22.367 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:31:22.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-214748364803 2024-09-10T20:31:22.368 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:31:22.369 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 214748364803 2024-09-10T20:31:22.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=214748364803 2024-09-10T20:31:22.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 214748364803' 2024-09-10T20:31:22.370 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:31:22.840 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 214748364804 -lt 214748364803 2024-09-10T20:31:22.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:31:22.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:31:22.841 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:31:23.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-10T20:31:23.432 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:31:23.432 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:31:23.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:31:23.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:31:23.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:31:23.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:31:23.433 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:31:23.849 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-10T20:31:23.849 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:31:23.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:31:23.850 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:31:24.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-10T20:31:24.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:31:24.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:31:24.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-10T20:31:24.454 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-10T20:31:24.454 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-10T20:31:24.454 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_122232 2024-09-10T20:31:24.454 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:31:24.454 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:31:25.454 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 84 68 0.265607 0.265625 0.17518 0.209562 2024-09-10T20:31:26.454 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 146 130 0.253877 0.242188 0.172833 0.233528 2024-09-10T20:31:27.457 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 200 184 0.239552 0.210938 0.225494 0.240511 2024-09-10T20:31:28.454 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 255 239 0.233366 0.214844 0.158429 0.254607 2024-09-10T20:31:29.455 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 326 310 0.242156 0.277344 0.242397 0.253092 2024-09-10T20:31:30.455 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 392 376 0.244759 0.257812 0.285012 0.250411 2024-09-10T20:31:31.455 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 458 442 0.246619 0.257812 0.117727 0.249132 2024-09-10T20:31:32.455 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 524 508 0.248015 0.257812 0.200635 0.246991 2024-09-10T20:31:33.455 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 585 569 0.246929 0.238281 0.329837 0.250158 2024-09-10T20:31:34.455 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 652 636 0.248404 0.261719 0.246591 0.247361 2024-09-10T20:31:35.455 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 712 696 0.247126 0.234375 0.355848 0.249052 2024-09-10T20:31:36.457 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 786 770 0.250618 0.289062 0.278255 0.248453 2024-09-10T20:31:37.456 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 845 829 0.249065 0.230469 0.175418 0.248013 2024-09-10T20:31:38.456 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 910 894 0.249408 0.253906 0.261301 0.24809 2024-09-10T20:31:39.456 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 966 950 0.247362 0.21875 0.12009 0.250519 2024-09-10T20:31:40.456 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 1037 1021 0.249234 0.277344 0.206052 0.248365 2024-09-10T20:31:41.456 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 1127 1111 0.255251 0.351562 0.200973 0.24387 2024-09-10T20:31:42.456 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 1208 1192 0.258647 0.316406 0.207473 0.239843 2024-09-10T20:31:43.456 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 1291 1275 0.262096 0.324219 0.225444 0.237052 2024-09-10T20:31:44.457 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:31:44.456600+0000 min lat: 0.114534 max lat: 0.485127 avg lat: 0.237166 2024-09-10T20:31:45.033 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:31:45.033 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 1359 1343 0.262271 0.265625 0.214287 0.237166 2024-09-10T20:31:45.457 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 1440 1424 0.264847 0.316406 0.134758 0.235142 2024-09-10T20:31:46.457 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 1518 1502 0.266657 0.304688 0.237008 0.233296 2024-09-10T20:31:47.457 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 1598 1582 0.268649 0.3125 0.200457 0.231841 2024-09-10T20:31:48.457 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 1682 1666 0.271125 0.328125 0.195839 0.229738 2024-09-10T20:31:49.457 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 1766 1750 0.273403 0.328125 0.125677 0.228052 2024-09-10T20:31:50.457 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 1852 1836 0.275806 0.335938 0.224491 0.225692 2024-09-10T20:31:51.457 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 1929 1913 0.27673 0.300781 0.184021 0.225 2024-09-10T20:31:52.457 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 2012 1996 0.278425 0.324219 0.172641 0.223831 2024-09-10T20:31:53.457 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 2090 2074 0.279329 0.304688 0.216228 0.222938 2024-09-10T20:31:54.458 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 2174 2158 0.280954 0.328125 0.16691 0.221555 2024-09-10T20:31:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.1963 2024-09-10T20:31:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 2174 2024-09-10T20:31:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-10T20:31:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-10T20:31:55.458 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.281232 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0406459 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.351562 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.210938 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 71 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 10.4053 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 90 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 54 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.22146 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0579782 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.485127 2024-09-10T20:31:55.459 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.0718697 2024-09-10T20:31:55.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 118134 2024-09-10T20:31:55.540 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-10T20:31:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 118134 2024-09-10T20:31:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (118134) - No such process 2024-09-10T20:31:56.542 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-10T20:31:57.112 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:31:57.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 119074 2024-09-10T20:31:57.131 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-10T20:31:58.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 119074 2024-09-10T20:31:58.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (119074) - No such process 2024-09-10T20:31:58.132 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-10T20:31:58.686 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:31:58.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 120015 2024-09-10T20:31:58.707 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-10T20:31:59.708 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 120015 2024-09-10T20:31:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (120015) - No such process 2024-09-10T20:31:59.709 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-10T20:32:00.183 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:32:00.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 120942 2024-09-10T20:32:00.201 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-10T20:32:01.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 120942 2024-09-10T20:32:01.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (120942) - No such process 2024-09-10T20:32:01.203 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-10T20:32:01.751 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 is already down. 2024-09-10T20:32:01.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:32:03.632 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:03.650 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-10T20:32:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records in 2024-09-10T20:32:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records out 2024-09-10T20:32:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.301298 s, 1.8 GB/s 2024-09-10T20:32:03.953 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-10T20:32:03.980 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:09.372 INFO:tasks.workunit.client.0.smithi039.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-10T20:32:09.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:32:11.748 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:11.767 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:32:13.648 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:13.664 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-10T20:32:14.239 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-10T20:32:14.239 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-10T20:32:14.239 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.573229 s, 1.9 GB/s 2024-09-10T20:32:14.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-10T20:32:14.264 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:22.904 INFO:tasks.workunit.client.0.smithi039.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-10T20:32:22.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-10T20:32:22.939 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:24.086 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:24.082+0000 7f1b2d3b3ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:24.878 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:24.874+0000 7f1b2d3b3ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:26.121 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:32:26.725 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:26.722+0000 7f0162804ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:27.516 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:27.514+0000 7f0162804ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:28.508 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:28.527 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:32:29.908 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:29.918 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-10T20:32:29.943 INFO:tasks.workunit.client.0.smithi039.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-10T20:32:29.943 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:31.292 INFO:tasks.workunit.client.0.smithi039.stdout:DB device added 2024-09-10T20:32:31.298 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-10T20:32:31.324 INFO:tasks.workunit.client.0.smithi039.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-10T20:32:31.324 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:32.169 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:32.166+0000 7ff28408dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:32.706 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:32.702+0000 7ff28408dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:32.777 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:32.774+0000 7ff28408dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:33.444 INFO:tasks.workunit.client.0.smithi039.stdout:WAL device added 2024-09-10T20:32:33.450 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-10T20:32:33.480 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:34.610 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:34.606+0000 7fafcc91dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:35.392 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:35.390+0000 7fafcc91dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:36.646 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:32:37.250 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:37.246+0000 7f82494f3ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:38.032 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:38.030+0000 7f82494f3ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:39.024 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:39.043 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:32:40.912 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:40.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-10T20:32:40.956 INFO:tasks.workunit.client.0.smithi039.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-10T20:32:40.956 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:43.328 INFO:tasks.workunit.client.0.smithi039.stdout:WAL device added 2024-09-10T20:32:43.333 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-10T20:32:43.361 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:32:46.513 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-10T20:32:46.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:32:48.396 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:32:48.415 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:32:48.416 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:32:48.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:32:48.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:32:48.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:32:48.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:32:48.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:32:48.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:32:48.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:32:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:32:48.421 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:32:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:32:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:32:48.422 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-10T20:32:48.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:32:48.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:32:48.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:32:48.424 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:32:48.458 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:48.453+0000 7ff53cda17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:48.461 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:48.457+0000 7ff53cda17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:48.462 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:48.457+0000 7ff53cda17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:48.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:32:48.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:32:48.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:32:48.915 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:32:48.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:32:48.916 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:32:48.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:48.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:32:48.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:48.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:49.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:50.015 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:50.009+0000 7ff53cda17c0 -1 Falling back to public interface 2024-09-10T20:32:50.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:50.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:50.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:32:50.380 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:32:50.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:50.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:50.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:51.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:51.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:51.861 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:32:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:32:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:51.861 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:52.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:53.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:53.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:53.306 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:32:53.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:32:53.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:53.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:53.746 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:53.886 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:53.881+0000 7ff53cda17c0 -1 osd.0 52 log_to_monitors true 2024-09-10T20:32:54.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:54.747 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:54.747 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:32:54.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:32:54.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:54.748 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:32:55.218 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 63 up_thru 63 down_at 53 last_clean_interval [38,52) [v2:127.0.0.1:6802/3098345779,v1:127.0.0.1:6803/3098345779] [v2:127.0.0.1:6804/3098345779,v1:127.0.0.1:6805/3098345779] exists,up 0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:32:55.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:32:55.218 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:32:55.219 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:32:55.219 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=128382 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:32:55.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:32:55.224 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:32:55.225 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:32:55.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:32:55.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:32:55.225 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-10T20:32:55.226 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:32:55.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:32:55.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:32:55.227 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-10T20:32:55.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:32:55.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:32:55.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:32:55.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:32:55.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:55.253+0000 7fc2b7a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:55.260 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:55.257+0000 7fc2b7a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:55.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:55.257+0000 7fc2b7a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:32:55.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:32:55.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:32:55.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:32:55.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:32:55.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:32:55.721 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:55.722 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:32:55.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:32:55.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:55.722 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:56.211 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:57.019 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:57.013+0000 7fc2b7a5c7c0 -1 Falling back to public interface 2024-09-10T20:32:57.212 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:57.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:57.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:32:57.213 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:32:57.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:57.213 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:57.642 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:57.803 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:57.797+0000 7fc2b7a5c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:58.587 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:58.581+0000 7fc2b7a5c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:32:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:32:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:32:58.644 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:32:58.644 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:32:58.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:32:58.645 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:32:59.108 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:32:59.603 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:32:59.597+0000 7fc2b7a5c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:33:00.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:00.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:00.109 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:33:00.110 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:33:00.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:00.110 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:33:00.408 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:00.401+0000 7fc2b7a5c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:33:00.630 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:01.132 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:01.125+0000 7fc2b7a5c7c0 -1 osd.1 55 log_to_monitors true 2024-09-10T20:33:01.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:01.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:01.632 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:33:01.632 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:33:01.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:01.633 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:33:02.174 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 66 up_thru 66 down_at 56 last_clean_interval [41,55) [v2:127.0.0.1:6810/675871780,v1:127.0.0.1:6811/675871780] [v2:127.0.0.1:6812/675871780,v1:127.0.0.1:6813/675871780] exists,up 5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:33:02.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:33:02.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:33:02.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:33:02.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-10T20:33:02.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=129353 2024-09-10T20:33:02.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-10T20:33:02.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:33:02.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:33:02.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:33:02.179 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:33:02.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:33:02.180 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:33:02.180 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:02.180 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:33:02.180 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:33:02.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-10T20:33:02.182 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:33:02.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:33:02.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:33:02.183 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-10T20:33:02.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:33:02.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:33:02.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:33:02.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:33:02.207 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:02.201+0000 7f7087f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:02.210 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:02.205+0000 7f7087f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:02.212 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:02.205+0000 7f7087f317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:02.652 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:33:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:33:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:33:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:33:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:33:02.653 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:02.653 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:33:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:33:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:33:02.654 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:03.125 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:04.127 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:33:04.128 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:33:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:04.128 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:33:04.195 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:04.189+0000 7f7087f317c0 -1 Falling back to public interface 2024-09-10T20:33:04.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:04.940 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:04.937+0000 7f7087f317c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:33:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:05.615 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:33:05.742 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:05.737+0000 7f7087f317c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:33:06.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:06.746 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:06.741+0000 7f7087f317c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:33:07.112 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:07.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:07.113 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:33:07.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:33:07.113 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:07.114 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:33:07.552 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:07.560 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:07.553+0000 7f7087f317c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:33:08.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:08.554 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:08.554 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:33:08.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:33:08.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:08.555 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:33:08.666 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:08.661+0000 7f7087f317c0 -1 osd.2 58 log_to_monitors true 2024-09-10T20:33:09.222 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 69 up_thru 56 down_at 59 last_clean_interval [45,58) [v2:127.0.0.1:6818/687348106,v1:127.0.0.1:6819/687348106] [v2:127.0.0.1:6820/687348106,v1:127.0.0.1:6821/687348106] exists,up 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:33:09.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:33:09.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:33:09.223 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:33:09.223 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=130323 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:33:09.226 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:33:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-10T20:33:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-10T20:33:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:33:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:33:09.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:33:09.228 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:33:09.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:33:09.230 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-10T20:33:09.231 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-10T20:33:09.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:33:09.231 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:33:09.231 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-10T20:33:09.232 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:33:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:33:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:33:09.233 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:33:09.258 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:09.253+0000 7f8f301697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:09.261 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:09.257+0000 7f8f301697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:09.263 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:09.257+0000 7f8f301697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:33:10.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:33:10.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:33:10.036 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:33:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:33:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:33:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:10.037 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:33:10.037 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:33:10.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:10.038 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:33:10.749 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:10.843 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:10.837+0000 7f8f301697c0 -1 Falling back to public interface 2024-09-10T20:33:11.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:11.751 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:11.751 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:33:11.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:33:11.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:11.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:33:12.227 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:13.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:13.228 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:13.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:33:13.229 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:33:13.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:13.229 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:33:13.691 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:33:14.378 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:33:14.373+0000 7f8f301697c0 -1 osd.3 60 log_to_monitors true 2024-09-10T20:33:14.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:33:14.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:33:14.693 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:33:14.693 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:33:14.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:33:14.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:33:15.294 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 74 up_thru 59 down_at 61 last_clean_interval [50,60) [v2:127.0.0.1:6826/4249527032,v1:127.0.0.1:6827/4249527032] [v2:127.0.0.1:6828/4249527032,v1:127.0.0.1:6829/4249527032] exists,up 35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:33:15.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:33:15.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:33:15.295 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:33:15.295 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-10T20:33:15.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=131293 2024-09-10T20:33:15.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-10T20:33:15.458 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-10T20:33:15.459 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-10T20:33:15.459 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_132218 2024-09-10T20:33:15.459 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:33:15.459 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:33:16.459 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 25 9 0.0351517 0.0351562 0.775041 0.526406 2024-09-10T20:33:17.460 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 61 45 0.0878584 0.140625 0.307682 0.67441 2024-09-10T20:33:18.459 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 112 96 0.124965 0.199219 0.349533 0.471101 2024-09-10T20:33:19.460 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 168 152 0.1484 0.21875 0.155179 0.401842 2024-09-10T20:33:20.460 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 228 212 0.165587 0.234375 0.302553 0.369787 2024-09-10T20:33:21.460 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 277 261 0.169886 0.191406 0.369411 0.357885 2024-09-10T20:33:22.460 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 329 313 0.174631 0.203125 0.364173 0.35135 2024-09-10T20:33:23.460 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 375 359 0.17526 0.179688 0.309155 0.349178 2024-09-10T20:33:24.460 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 428 412 0.178787 0.207031 0.274393 0.345342 2024-09-10T20:33:25.460 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 475 459 0.179265 0.183594 0.274932 0.34193 2024-09-10T20:33:26.461 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 525 509 0.180721 0.195312 0.245432 0.339336 2024-09-10T20:33:27.461 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 569 553 0.179982 0.171875 0.335646 0.341583 2024-09-10T20:33:28.461 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 617 601 0.180558 0.1875 0.205895 0.341223 2024-09-10T20:33:29.461 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 671 655 0.182726 0.210938 0.291996 0.337273 2024-09-10T20:33:30.461 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 720 704 0.183303 0.191406 0.230137 0.337024 2024-09-10T20:33:31.461 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 773 757 0.184784 0.207031 0.199791 0.334403 2024-09-10T20:33:32.461 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 809 793 0.182185 0.140625 0.368158 0.339458 2024-09-10T20:33:33.462 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 864 848 0.183998 0.214844 0.195773 0.336967 2024-09-10T20:33:34.465 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 910 894 0.18377 0.179688 0.307624 0.336035 2024-09-10T20:33:35.462 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:33:35.461399+0000 min lat: 0.155179 max lat: 1.89782 avg lat: 0.33578 2024-09-10T20:33:35.462 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:33:35.462 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 962 946 0.184736 0.203125 0.317874 0.33578 2024-09-10T20:33:36.462 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 1013 997 0.185425 0.199219 0.335893 0.334683 2024-09-10T20:33:37.462 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 1057 1041 0.184808 0.171875 0.311536 0.335643 2024-09-10T20:33:38.462 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 1109 1093 0.185603 0.203125 0.27649 0.334417 2024-09-10T20:33:39.462 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 1158 1142 0.185844 0.191406 0.213766 0.334695 2024-09-10T20:33:40.462 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 1209 1193 0.186378 0.199219 0.385015 0.334137 2024-09-10T20:33:41.462 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 1255 1239 0.18612 0.179688 0.283423 0.333689 2024-09-10T20:33:42.463 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 1295 1279 0.185013 0.15625 0.452771 0.334578 2024-09-10T20:33:43.463 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 1344 1328 0.18524 0.191406 0.367527 0.334801 2024-09-10T20:33:44.463 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 1395 1379 0.185721 0.199219 0.304862 0.333851 2024-09-10T20:33:45.463 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 1446 1430 0.186171 0.199219 0.333123 0.334038 2024-09-10T20:33:46.463 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.2 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 1446 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.187035 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0352285 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.234375 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.0351562 2024-09-10T20:33:46.464 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 47 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 9.0185 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 60 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 9 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.333455 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.12112 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 1.89782 2024-09-10T20:33:46.465 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.155179 2024-09-10T20:33:46.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128382 2024-09-10T20:33:46.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-10T20:33:47.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128382 2024-09-10T20:33:47.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (128382) - No such process 2024-09-10T20:33:47.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-10T20:33:48.169 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:33:48.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129353 2024-09-10T20:33:48.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-10T20:33:49.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129353 2024-09-10T20:33:49.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (129353) - No such process 2024-09-10T20:33:49.194 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-10T20:33:49.662 INFO:tasks.workunit.client.0.smithi039.stderr:osd.1 is already down. 2024-09-10T20:33:49.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 130323 2024-09-10T20:33:49.681 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-10T20:33:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 130323 2024-09-10T20:33:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (130323) - No such process 2024-09-10T20:33:50.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-10T20:33:51.158 INFO:tasks.workunit.client.0.smithi039.stderr:osd.2 is already down. 2024-09-10T20:33:51.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 131293 2024-09-10T20:33:51.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-10T20:33:52.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 131293 2024-09-10T20:33:52.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (131293) - No such process 2024-09-10T20:33:52.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-10T20:33:52.751 INFO:tasks.workunit.client.0.smithi039.stderr:osd.3 is already down. 2024-09-10T20:33:52.768 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:33:55.133 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:33:55.157 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-10T20:33:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-10T20:33:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-10T20:33:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.572916 s, 1.9 GB/s 2024-09-10T20:33:55.732 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-10T20:33:55.758 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:34:05.921 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-10T20:34:05.921 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-10T20:34:05.927 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:34:08.305 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:08.325 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-10T20:34:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records in 2024-09-10T20:34:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:512+0 records out 2024-09-10T20:34:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.304413 s, 1.8 GB/s 2024-09-10T20:34:08.631 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-10T20:34:08.655 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:34:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-10T20:34:14.809 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-10T20:34:14.817 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-10T20:34:14.887 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-10T20:34:17.257 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:17.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:34:17.892 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:17.887+0000 7f75de6d1ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:18.678 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:18.675+0000 7f75de6d1ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:19.653 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:19.669 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-10T20:34:20.238 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-10T20:34:20.239 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-10T20:34:20.239 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.5683 s, 1.9 GB/s 2024-09-10T20:34:20.239 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-10T20:34:20.263 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:34:21.410 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:21.407+0000 7f590863dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:22.193 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:22.191+0000 7f590863dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:28.736 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:28.731+0000 7f590863dac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:30.193 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-10T20:34:30.193 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-10T20:34:30.193 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-10T20:34:30.202 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-10T20:34:30.320 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:34:30.926 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:30.923+0000 7fa0a8c17ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:31.469 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:31.467+0000 7fa0a8c17ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:32.205 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:32.221 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-10T20:34:32.249 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:34:33.102 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:33.099+0000 7f17db562ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:33.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:33.627+0000 7f17db562ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:34.389 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-10T20:34:34.399 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-10T20:34:34.523 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-10T20:34:35.901 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:35.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-10T20:34:38.013 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:38.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-10T20:34:38.601 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-10T20:34:38.602 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-10T20:34:38.602 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.570877 s, 1.9 GB/s 2024-09-10T20:34:38.602 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-10T20:34:38.627 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:34:39.481 INFO:tasks.workunit.client.0.smithi039.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-10T20:34:39.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-10T20:34:39.485 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:34:40.082 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:40.078+0000 7fe444438ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:40.865 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:40.862+0000 7fe444438ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:41.857 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:41.876 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-10T20:34:48.841 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-10T20:34:48.841 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-10T20:34:48.841 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.82721 s, 157 MB/s 2024-09-10T20:34:48.841 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-10T20:34:48.869 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:34:50.007 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:50.002+0000 7f6996c26ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:50.789 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:50.786+0000 7f6996c26ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:50.863 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:50.858+0000 7f6996c26ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:52.033 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-10T20:34:52.033 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-10T20:34:52.033 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-10T20:34:52.042 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-10T20:34:52.659 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:52.654+0000 7f87a21eeac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:53.189 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:34:53.182+0000 7f87a21eeac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:34:53.933 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:34:53.951 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-10T20:34:54.525 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records in 2024-09-10T20:34:54.525 INFO:tasks.workunit.client.0.smithi039.stderr:1024+0 records out 2024-09-10T20:34:54.525 INFO:tasks.workunit.client.0.smithi039.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.572491 s, 1.9 GB/s 2024-09-10T20:34:54.525 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-10T20:34:54.542 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:35:03.685 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-10T20:35:03.685 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-10T20:35:03.686 INFO:tasks.workunit.client.0.smithi039.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-10T20:35:03.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-10T20:35:04.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:04.322+0000 7f5d0051bac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:04.867 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:04.862+0000 7f5d0051bac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:05.593 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:35:05.612 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:05.613 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:35:05.614 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:35:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:35:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:05.615 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:35:05.616 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:35:05.616 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:35:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:35:05.617 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:35:05.617 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-10T20:35:05.618 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:35:05.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:05.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:05.619 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:05.651 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:05.646+0000 7f5e28bf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:05.654 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:05.650+0000 7f5e28bf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:05.655 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:05.650+0000 7f5e28bf17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:06.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:35:06.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:06.095 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:35:06.096 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:35:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:06.096 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:06.097 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:06.568 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:07.376 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:07.370+0000 7f5e28bf17c0 -1 Falling back to public interface 2024-09-10T20:35:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:07.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:07.570 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:35:07.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:07.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:07.571 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:08.047 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:09.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:09.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:09.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:09.049 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:35:09.049 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:09.050 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:09.494 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:10.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:10.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:10.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:35:10.496 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:35:10.496 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:10.497 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:10.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:11.442 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:11.438+0000 7f5e28bf17c0 -1 osd.0 77 log_to_monitors true 2024-09-10T20:35:11.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:11.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:11.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:35:11.980 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:35:11.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:11.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:35:12.430 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 88 up_thru 88 down_at 78 last_clean_interval [63,77) [v2:127.0.0.1:6802/2131170575,v1:127.0.0.1:6803/2131170575] [v2:127.0.0.1:6804/2131170575,v1:127.0.0.1:6805/2131170575] exists,up 0618f729-b530-4370-b507-bc57485ba1dd 2024-09-10T20:35:12.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:12.430 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:12.431 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:12.431 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:35:12.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=138395 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=1 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-10T20:35:12.435 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:12.436 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-10T20:35:12.437 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.1 2024-09-10T20:35:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.1 2024-09-10T20:35:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 1 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:35:12.438 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-10T20:35:12.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 1 = 1 ']' 2024-09-10T20:35:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:12.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:12.471 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:12.466+0000 7f971136f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:12.473 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:12.470+0000 7f971136f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:12.475 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:12.470+0000 7f971136f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:12.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 1 2024-09-10T20:35:12.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:12.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:35:12.908 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:12.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:12.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:12.909 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:35:12.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:12.909 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:12.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:13.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:13.716 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:13.710+0000 7f971136f7c0 -1 Falling back to public interface 2024-09-10T20:35:14.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:14.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:14.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:14.378 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:35:14.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:14.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:14.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:15.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:15.807 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:15.808 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:35:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:15.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:35:15.896 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:15.890+0000 7f971136f7c0 -1 osd.1 80 log_to_monitors true 2024-09-10T20:35:16.328 INFO:tasks.workunit.client.0.smithi039.stdout:osd.1 up in weight 1 up_from 91 up_thru 78 down_at 81 last_clean_interval [66,80) [v2:127.0.0.1:6810/871516235,v1:127.0.0.1:6811/871516235] [v2:127.0.0.1:6812/871516235,v1:127.0.0.1:6813/871516235] exists,up 5569bcd7-ae19-4a16-91c5-cb40384eedda 2024-09-10T20:35:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:16.329 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:16.329 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-10T20:35:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=139366 2024-09-10T20:35:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-10T20:35:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:35:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=2 2024-09-10T20:35:16.330 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:16.331 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:35:16.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:16.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:16.332 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:16.333 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:16.334 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:35:16.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-10T20:35:16.335 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.2 2024-09-10T20:35:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.2 2024-09-10T20:35:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 2 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:35:16.336 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-10T20:35:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 2 = 2 ']' 2024-09-10T20:35:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:16.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:16.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:16.366 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:16.362+0000 7f15fd5f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:16.469 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:16.466+0000 7f15fd5f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:16.470 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:16.466+0000 7f15fd5f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 2 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:16.888 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:35:16.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:16.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:16.889 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:17.489 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:18.000 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:17.994+0000 7f15fd5f17c0 -1 Falling back to public interface 2024-09-10T20:35:18.490 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:18.491 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:18.491 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:35:18.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:18.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:18.492 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:18.776 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:18.770+0000 7f15fd5f17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:18.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:19.307 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:19.302+0000 7f15fd5f17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:19.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:19.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:19.983 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:35:19.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:19.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:19.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:20.095 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:20.090+0000 7f15fd5f17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:20.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:20.643 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:20.638+0000 7f15fd5f17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:21.074 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:21.069+0000 7f15fd5f17c0 -1 osd.2 83 log_to_monitors true 2024-09-10T20:35:21.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:21.438 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:21.439 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:35:21.439 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:35:21.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:21.440 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:35:22.293 INFO:tasks.workunit.client.0.smithi039.stdout:osd.2 up in weight 1 up_from 95 up_thru 95 down_at 84 last_clean_interval [69,83) [v2:127.0.0.1:6818/1487314804,v1:127.0.0.1:6819/1487314804] [v2:127.0.0.1:6820/1487314804,v1:127.0.0.1:6821/1487314804] exists,up 773ff91d-32bb-49f0-b7cc-3a7164b0ec86 2024-09-10T20:35:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:22.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:22.294 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-10T20:35:22.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=140293 2024-09-10T20:35:22.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=3 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:35:22.305 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:22.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:35:22.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-10T20:35:22.308 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.3 2024-09-10T20:35:22.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.3 2024-09-10T20:35:22.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 3 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:35:22.309 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-10T20:35:22.309 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 3 = 3 ']' 2024-09-10T20:35:22.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:35:22.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:35:22.310 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:35:22.338 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:22.333+0000 7f8ca18967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:22.340 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:22.333+0000 7f8ca18967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:22.342 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:22.337+0000 7f8ca18967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:35:22.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 3 2024-09-10T20:35:22.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:35:22.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:35:22.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:35:22.882 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:35:22.882 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:35:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:35:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:22.883 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:35:23.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:24.080 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:24.073+0000 7f8ca18967c0 -1 Falling back to public interface 2024-09-10T20:35:24.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:24.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:24.376 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:35:24.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:35:24.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:24.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:35:24.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:25.092 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:25.085+0000 7f8ca18967c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:25.648 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:25.641+0000 7f8ca18967c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:25.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:25.867 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:25.868 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:35:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:35:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:25.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:35:26.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:35:26.455 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:26.449+0000 7f8ca18967c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:26.998 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:26.993+0000 7f8ca18967c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-09-10T20:35:27.353 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:35:27.345+0000 7f8ca18967c0 -1 osd.3 85 log_to_monitors true 2024-09-10T20:35:27.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:35:27.377 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:35:27.377 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:35:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:35:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:35:27.378 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:35:27.970 INFO:tasks.workunit.client.0.smithi039.stdout:osd.3 up in weight 1 up_from 100 up_thru 84 down_at 86 last_clean_interval [74,85) [v2:127.0.0.1:6826/3956975097,v1:127.0.0.1:6827/3956975097] [v2:127.0.0.1:6828/3956975097,v1:127.0.0.1:6829/3956975097] exists,up 35fcc31c-2a74-4f0c-8e4d-df88d69846db 2024-09-10T20:35:27.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:35:27.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:35:27.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:35:27.971 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-10T20:35:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=141216 2024-09-10T20:35:27.973 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-10T20:35:28.004 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-10T20:35:28.004 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-10T20:35:28.004 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_142141 2024-09-10T20:35:28.005 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:35:28.005 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:35:29.004 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 36 20 0.078117 0.078125 0.14559 0.389664 2024-09-10T20:35:30.004 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 157 141 0.275357 0.472656 0.0958655 0.218597 2024-09-10T20:35:31.005 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 253 237 0.308554 0.375 0.168889 0.195301 2024-09-10T20:35:32.005 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 365 349 0.34078 0.4375 0.134633 0.180714 2024-09-10T20:35:33.005 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 461 445 0.347576 0.375 0.167718 0.17551 2024-09-10T20:35:34.005 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 601 585 0.380779 0.546875 0.111212 0.162165 2024-09-10T20:35:35.006 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 707 691 0.385527 0.414062 0.134237 0.159996 2024-09-10T20:35:36.006 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 819 803 0.392017 0.4375 0.14757 0.158132 2024-09-10T20:35:37.007 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 937 921 0.399669 0.460938 0.103745 0.155337 2024-09-10T20:35:38.006 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 1059 1043 0.407353 0.476562 0.12791 0.152206 2024-09-10T20:35:39.006 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 1161 1145 0.406539 0.398438 0.174783 0.15186 2024-09-10T20:35:40.006 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 1292 1276 0.415299 0.511719 0.106699 0.150012 2024-09-10T20:35:41.006 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 1373 1357 0.407689 0.316406 0.116332 0.152867 2024-09-10T20:35:42.007 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 1485 1469 0.409814 0.4375 0.159648 0.151605 2024-09-10T20:35:43.008 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 1597 1581 0.411655 0.4375 0.126273 0.151153 2024-09-10T20:35:44.007 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 1719 1703 0.415707 0.476562 0.133286 0.149649 2024-09-10T20:35:45.008 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 1825 1809 0.415586 0.414062 0.142578 0.149621 2024-09-10T20:35:46.008 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 1905 1889 0.409856 0.3125 0.182666 0.151544 2024-09-10T20:35:47.008 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 2018 2002 0.411514 0.441406 0.128695 0.151482 2024-09-10T20:35:48.008 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:35:48.007156+0000 min lat: 0.0685835 max lat: 1.08106 avg lat: 0.150084 2024-09-10T20:35:48.008 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:35:48.008 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 2136 2120 0.413982 0.460938 0.128404 0.150084 2024-09-10T20:35:49.008 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 2232 2216 0.412123 0.375 0.166888 0.150941 2024-09-10T20:35:50.008 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 2354 2338 0.41505 0.476562 0.144292 0.150274 2024-09-10T20:35:51.009 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 2437 2421 0.4111 0.324219 0.245764 0.151013 2024-09-10T20:35:52.009 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 2549 2533 0.412198 0.4375 0.124114 0.151221 2024-09-10T20:35:53.009 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 2661 2645 0.413208 0.4375 0.125709 0.150804 2024-09-10T20:35:54.009 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 2776 2760 0.414591 0.449219 0.139517 0.150243 2024-09-10T20:35:55.009 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 2874 2858 0.413412 0.382812 0.109679 0.150691 2024-09-10T20:35:56.009 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 3018 3002 0.418734 0.5625 0.0919422 0.149011 2024-09-10T20:35:57.009 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 3083 3067 0.413049 0.253906 0.173151 0.150995 2024-09-10T20:35:58.009 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 3195 3179 0.413863 0.4375 0.128855 0.150586 2024-09-10T20:35:59.009 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.1068 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 3195 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.41454 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0922535 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.5625 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.078125 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 106 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 23.6169 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 144 2024-09-10T20:35:59.010 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 20 2024-09-10T20:35:59.011 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.150538 2024-09-10T20:35:59.011 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0646733 2024-09-10T20:35:59.011 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 1.08106 2024-09-10T20:35:59.011 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.0685835 2024-09-10T20:35:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-10T20:35:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:35:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:35:59.074 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:35:59.075 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:35:59.076 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:35:59.076 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:35:59.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:35:59.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:35:59.076 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:35:59.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:35:59.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:35:59.204 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:35:59.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:35:59.205 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:35:59.205 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:35:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:35:59.694 INFO:tasks.workunit.client.0.smithi039.stderr:1 2024-09-10T20:35:59.695 INFO:tasks.workunit.client.0.smithi039.stderr:2 2024-09-10T20:35:59.695 INFO:tasks.workunit.client.0.smithi039.stderr:3' 2024-09-10T20:35:59.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:35:59.695 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:59.696 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:35:59.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=377957122060 2024-09-10T20:35:59.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 377957122060 2024-09-10T20:35:59.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122060' 2024-09-10T20:35:59.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:35:59.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:36:00.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=390842023947 2024-09-10T20:36:00.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 390842023947 2024-09-10T20:36:00.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122060 1-390842023947' 2024-09-10T20:36:00.054 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:00.054 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:36:00.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=408021893130 2024-09-10T20:36:00.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 408021893130 2024-09-10T20:36:00.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122060 1-390842023947 2-408021893130' 2024-09-10T20:36:00.210 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:36:00.211 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:36:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=429496729609 2024-09-10T20:36:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 429496729609 2024-09-10T20:36:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-377957122060 1-390842023947 2-408021893130 3-429496729609' 2024-09-10T20:36:00.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:00.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-377957122060 2024-09-10T20:36:00.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:00.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:36:00.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-377957122060 2024-09-10T20:36:00.372 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:00.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=377957122060 2024-09-10T20:36:00.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 377957122060' 2024-09-10T20:36:00.374 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 377957122060 2024-09-10T20:36:00.374 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:36:00.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 377957122060 -lt 377957122060 2024-09-10T20:36:00.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:00.863 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-390842023947 2024-09-10T20:36:00.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:00.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:36:00.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-390842023947 2024-09-10T20:36:00.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:00.868 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.1 seq 390842023947 2024-09-10T20:36:00.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=390842023947 2024-09-10T20:36:00.868 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 390842023947' 2024-09-10T20:36:00.868 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:36:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 390842023947 -lt 390842023947 2024-09-10T20:36:01.414 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:01.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-408021893130 2024-09-10T20:36:01.415 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:01.417 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:36:01.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-408021893130 2024-09-10T20:36:01.418 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:01.419 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=408021893130 2024-09-10T20:36:01.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 408021893130' 2024-09-10T20:36:01.420 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.2 seq 408021893130 2024-09-10T20:36:01.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:36:01.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 408021893130 -lt 408021893130 2024-09-10T20:36:01.904 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:36:01.905 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-429496729609 2024-09-10T20:36:01.906 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:36:01.907 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:36:01.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-429496729609 2024-09-10T20:36:01.909 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:36:01.910 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=429496729609 2024-09-10T20:36:01.910 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.3 seq 429496729609 2024-09-10T20:36:01.911 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 429496729609' 2024-09-10T20:36:01.911 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:36:02.368 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729607 -lt 429496729609 2024-09-10T20:36:02.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:36:03.369 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:36:03.371 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:36:03.806 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 429496729609 -lt 429496729609 2024-09-10T20:36:03.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:36:03.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:36:03.807 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:36:04.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-10T20:36:04.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:36:04.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:36:04.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:36:04.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:36:04.390 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:36:04.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:36:04.391 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:36:04.804 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-10T20:36:04.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:36:04.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:36:04.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:36:05.362 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-10T20:36:05.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:36:05.363 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:36:05.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:36:05.363 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:36:05.364 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:36:05.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:36:05.676 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:36:05.676 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:36:05.677 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:36:05.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:36:05.678 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:36:05.679 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:36:05.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:05.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:36:05.680 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:36:05.680 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:05.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:36:05.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:36:05.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-10T20:36:06.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:36:06.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:06.167 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:06.168 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.95628 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:36:06.170 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-10T20:36:06.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:36:06.171 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:36:06.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:36:06.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:36:06.171 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:36:06.174 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:36:06.174 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:36:06.175 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:36:06.176 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:36:06.176 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:36:06.177 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:36:06.177 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:36:06.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:06.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:36:06.178 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:36:06.178 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:36:06.179 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:36:06.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:36:06.180 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-10T20:36:06.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:36:06.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:06.182 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:06.182 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.95628 2024-09-10T20:36:06.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:36:06.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:36:06.183 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-10T20:36:06.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:36:06.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:06.185 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:06.185 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.95628 2024-09-10T20:36:06.187 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:36:06.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:36:06.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:36:06.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-10T20:36:06.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-10T20:36:06.187 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:06.188 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-10T20:36:06.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-10T20:36:06.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-10T20:36:06.188 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-10T20:36:06.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-10T20:36:06.189 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-10T20:36:06.189 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-10T20:36:06.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none ' 2024-09-10T20:36:06.190 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:36:06.191 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:36:06.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:36:06.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-10T20:36:06.192 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:06.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:36:06.659 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:06.660 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:06.660 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:06.660 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:06.660 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:06.661 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:36:06.661 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:36:06.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:36:06.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:36:06.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:36:06.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:36:06.864 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:36:06.865 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:36:06.865 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:36:06.865 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:36:06.865 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:36:06.866 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:36:06.866 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:06.866 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:06.866 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.95628/ceph-mon.a.asok 2024-09-10T20:36:06.866 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:36:06.867 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.95628/ceph-mon.a.asok config get fsid 2024-09-10T20:36:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:36:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:36:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:36:06.984 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:36:06.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:36:06.985 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.95628/ceph-mon.a.asok 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:36:06.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.95628/ceph-mon.a.asok config get mon_host 2024-09-10T20:36:07.102 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-10T20:36:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:36:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:36:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:36:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-10T20:36:07.103 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:36:08.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:36:08.022 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:08.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:08.023 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:08.023 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:08.023 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:08.023 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:36:08.024 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:36:08.026 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:36:08.087 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:36:08.088 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:36:08.089 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:36:08.090 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:36:08.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2324cb2-9891-4aa8-a784-f75c876839d6 2024-09-10T20:36:08.091 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f2324cb2-9891-4aa8-a784-f75c876839d6' 2024-09-10T20:36:08.091 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 f2324cb2-9891-4aa8-a784-f75c876839d6 2024-09-10T20:36:08.091 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:36:08.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC4reBmLPl2BhAA11oycE6YusEL+VNv6G9ocQ== 2024-09-10T20:36:08.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC4reBmLPl2BhAA11oycE6YusEL+VNv6G9ocQ=="}' 2024-09-10T20:36:08.111 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2324cb2-9891-4aa8-a784-f75c876839d6 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-10T20:36:08.418 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:36:08.433 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-10T20:36:08.434 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4reBmLPl2BhAA11oycE6YusEL+VNv6G9ocQ== --osd-uuid f2324cb2-9891-4aa8-a784-f75c876839d6 2024-09-10T20:36:08.468 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:08.460+0000 7facb1cc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:08.475 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:08.468+0000 7facb1cc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:08.476 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:08.468+0000 7facb1cc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:08.472+0000 7facb1cc87c0 -1 bdev(0x558fce7de800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:36:08.477 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:08.472+0000 7facb1cc87c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-10T20:36:16.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-10T20:36:16.294 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:36:16.296 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:36:16.296 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:36:16.297 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:36:16.892 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:36:16.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:36:16.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:36:16.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:36:16.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:36:16.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:36:16.931 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:16.924+0000 7f185f5067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:16.933 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:16.928+0000 7f185f5067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:16.934 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:16.928+0000 7f185f5067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:17.375 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:36:17.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:36:17.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:17.376 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:17.860 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:18.452 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:18.444+0000 7f185f5067c0 -1 Falling back to public interface 2024-09-10T20:36:18.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:18.862 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:18.863 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:36:18.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:36:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:18.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:19.335 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:20.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:20.336 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:20.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:36:20.337 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:36:20.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:20.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:20.823 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:21.724 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:36:21.716+0000 7f185f5067c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:36:21.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:21.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:21.825 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:36:21.825 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:36:21.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:21.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:22.968 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:23.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:23.970 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:23.970 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:36:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:36:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:23.971 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:24.808 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:36:25.809 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:36:25.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:36:25.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:36:25.810 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:36:25.810 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:36:25.811 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:36:26.282 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2172120433,v1:127.0.0.1:6803/2172120433] [v2:127.0.0.1:6804/2172120433,v1:127.0.0.1:6805/2172120433] exists,up f2324cb2-9891-4aa8-a784-f75c876839d6 2024-09-10T20:36:26.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:36:26.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:36:26.282 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:36:26.283 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:36:26.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=144041 2024-09-10T20:36:26.284 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-10T20:36:31.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-10T20:36:31.286 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-10T20:36:31.838 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-10T20:36:31.855 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:36:32.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-10T20:36:32.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-10T20:36:32.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-10T20:36:32.858 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-10T20:36:32.904 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-10T20:36:32.906 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-10T20:36:32.906 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_145298 2024-09-10T20:36:32.941 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:36:32.941 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:36:33.941 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 33 17 67.9922 68 0.818573 0.843049 2024-09-10T20:36:34.941 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 49 33 65.991 64 0.81027 0.823718 2024-09-10T20:36:35.941 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 65 49 65.324 64 0.960655 0.867974 2024-09-10T20:36:36.941 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 81 65 64.99 64 0.93554 0.884857 2024-09-10T20:36:37.942 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 97 81 64.79 64 1.13649 0.917744 2024-09-10T20:36:38.942 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 113 97 64.6566 64 0.885178 0.913489 2024-09-10T20:36:39.942 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 129 113 64.5615 64 1.05248 0.943908 2024-09-10T20:36:40.942 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 145 129 64.49 64 0.951935 0.94521 2024-09-10T20:36:41.942 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 161 145 64.4345 64 0.927621 0.943224 2024-09-10T20:36:42.942 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 177 161 64.3898 64 0.796886 0.929027 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 10.6828 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 177 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4194304 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4194304 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 66.2748 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 1.26491 2024-09-10T20:36:43.943 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 68 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 64 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 16 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 0.316228 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 17 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 16 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.960068 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.147184 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 1.27772 2024-09-10T20:36:43.944 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.150112 2024-09-10T20:36:43.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-10T20:36:53.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-10T20:36:53.999 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-10T20:36:54.181 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1063256064 2024-09-10T20:36:54.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-10T20:36:54.181 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-10T20:36:54.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=147456000 2024-09-10T20:36:54.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-10T20:36:54.352 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 147456000 -eq 0 2024-09-10T20:36:54.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-10T20:36:54.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 147456000 -gt 0 2024-09-10T20:36:54.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 144041 2024-09-10T20:36:54.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-10T20:36:55.353 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 144041 2024-09-10T20:36:55.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (144041) - No such process 2024-09-10T20:36:55.354 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-10T20:36:55.840 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:36:55.856 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-10T20:36:55.886 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:37:06.517 INFO:tasks.workunit.client.0.smithi039.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-10T20:37:06.536 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-10T20:37:06.570 INFO:tasks.workunit.client.0.smithi039.stdout:inferring bluefs devices from bluestore path 2024-09-10T20:37:08.631 INFO:tasks.workunit.client.0.smithi039.stdout:1 : device size 0xffffe000 : using 0x484b8000(1.1 GiB) 2024-09-10T20:37:08.637 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-10T20:37:11.537 INFO:tasks.workunit.client.0.smithi039.stdout:fsck success 2024-09-10T20:37:11.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:37:11.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:11.586 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:37:11.587 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:37:11.589 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:37:11.590 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:37:11.591 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:37:11.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:37:11.591 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:37:11.591 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-10T20:37:11.592 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:37:11.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:37:11.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:37:11.593 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:37:11.629 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:11.623+0000 7f56f8eb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:11.632 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:11.627+0000 7f56f8eb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:11.633 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:11.631+0000 7f56f8eb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:12.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:37:12.029 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:37:12.030 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:12.031 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:12.506 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:12.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:12.883+0000 7f56f8eb37c0 -1 Falling back to public interface 2024-09-10T20:37:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:13.508 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:13.508 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:37:13.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:37:13.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:13.509 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:14.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:15.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:15.017 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:15.017 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:37:15.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:37:15.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:15.018 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:15.461 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:16.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:16.463 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:16.463 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:37:16.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:37:16.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:16.464 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:16.933 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:17.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:17.935 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:17.935 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:37:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:37:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:17.936 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:18.379 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:19.380 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:19.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:19.381 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:37:19.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:37:19.381 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:19.382 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:19.411 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:19.407+0000 7f56f8eb37c0 -1 osd.0 10 log_to_monitors true 2024-09-10T20:37:20.470 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:21.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:21.471 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:21.472 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:37:21.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:37:21.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:21.472 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:22.371 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:23.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:23.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:23.373 INFO:tasks.workunit.client.0.smithi039.stdout:7 2024-09-10T20:37:23.373 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:37:23.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:23.374 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:23.891 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/573325708,v1:127.0.0.1:6803/573325708] [v2:127.0.0.1:6804/573325708,v1:127.0.0.1:6805/573325708] exists,up f2324cb2-9891-4aa8-a784-f75c876839d6 2024-09-10T20:37:23.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:37:23.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:37:23.891 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:37:23.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:37:23.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=146529 2024-09-10T20:37:23.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-10T20:37:23.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:37:23.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:37:23.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:37:23.894 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:37:23.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:37:23.895 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:37:23.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:37:23.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:37:23.895 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:37:24.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:37:24.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:37:24.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:37:24.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:37:24.015 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:37:24.015 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:37:24.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids=0 2024-09-10T20:37:24.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:37:24.511 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:37:24.512 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:37:24.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=55834574852 2024-09-10T20:37:24.683 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 55834574852 2024-09-10T20:37:24.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-55834574852' 2024-09-10T20:37:24.684 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:37:24.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:37:24.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-55834574852 2024-09-10T20:37:24.685 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:37:24.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-55834574852 2024-09-10T20:37:24.686 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:37:24.686 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=55834574852 2024-09-10T20:37:24.687 INFO:tasks.workunit.client.0.smithi039.stdout:waiting osd.0 seq 55834574852 2024-09-10T20:37:24.687 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-10T20:37:24.687 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:37:25.142 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-10T20:37:25.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:37:25.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:37:25.143 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:37:25.752 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 16 == 0 2024-09-10T20:37:25.753 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:37:25.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:37:25.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:37:25.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:37:25.753 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:37:25.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:37:25.754 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:37:26.197 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=16 2024-09-10T20:37:26.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:37:26.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:37:26.198 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:37:26.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 = 16 2024-09-10T20:37:26.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:37:26.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:37:26.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-10T20:37:26.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:26.780 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:37:26.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-10T20:37:26.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:37:26.781 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:37:26.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:37:26.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:37:26.781 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:37:26.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:37:26.890 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:37:26.890 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:37:26.891 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:37:26.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:37:26.892 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:37:26.892 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:37:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:37:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:37:26.893 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:37:26.893 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:37:26.894 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:37:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:37:26.895 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-10T20:37:26.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:37:26.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:26.976 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:26.976 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.95628 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-bluefs-volume-ops 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:37:26.978 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:37:26.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:37:26.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:37:26.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:37:26.979 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:37:26.979 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:37:26.980 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:37:26.980 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:37:26.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:37:26.981 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:37:26.981 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:37:26.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:37:26.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:37:26.982 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:37:26.982 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:37:26.983 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:37:26.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:37:26.984 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-10T20:37:26.985 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:37:26.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:26.986 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:26.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.95628 2024-09-10T20:37:26.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:37:26.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:37:26.986 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-10T20:37:26.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:37:26.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:26.987 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:26.987 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.95628 2024-09-10T20:37:26.989 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:37:26.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T20:37:26.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:37:26.989 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-10T20:37:26.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-10T20:37:26.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:26.990 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-10T20:37:26.990 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-10T20:37:26.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-10T20:37:26.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-10T20:37:26.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-10T20:37:26.991 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-10T20:37:26.991 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-10T20:37:26.992 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=4115e0c7-00a9-4f28-97ce-bb9c8ce7350f --auth-supported=none ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-10T20:37:26.993 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:26.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:37:26.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:37:26.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:37:26.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-10T20:37:26.994 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:27.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:37:27.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:37:27.444 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:37:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:37:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:27.445 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:37:27.445 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:37:27.682 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:37:27.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:37:27.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:37:27.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:37:27.683 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:37:27.684 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:37:27.684 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:37:27.684 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.95628/ceph-mon.a.asok 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:37:27.685 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.95628/ceph-mon.a.asok config get fsid 2024-09-10T20:37:27.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:37:27.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:37:27.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:37:27.804 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:37:27.805 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:37:27.805 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:37:27.805 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:37:27.805 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:37:27.805 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:37:27.806 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:27.806 INFO:tasks.workunit.client.0.smithi039.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:27.806 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.95628/ceph-mon.a.asok 2024-09-10T20:37:27.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:37:27.806 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.95628/ceph-mon.a.asok config get mon_host 2024-09-10T20:37:27.916 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-10T20:37:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:37:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:37:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:37:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-10T20:37:27.917 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:37:28.854 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:37:28.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:37:28.855 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:37:28.855 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:37:28.855 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:28.856 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:28.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:37:28.856 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:37:28.857 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:37:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-10T20:37:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:37:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:37:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:37:28.920 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=4115e0c7-00a9-4f28-97ce-bb9c8ce7350f --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:37:28.921 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:37:28.922 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b1980db4-01ed-4478-bc7d-4914bae05557 2024-09-10T20:37:28.923 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b1980db4-01ed-4478-bc7d-4914bae05557' 2024-09-10T20:37:28.924 INFO:tasks.workunit.client.0.smithi039.stdout:add osd0 b1980db4-01ed-4478-bc7d-4914bae05557 2024-09-10T20:37:28.924 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:37:28.948 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIruBmJfRfOBAA5N8KV3c+cLDhcd8GmF2+Jg== 2024-09-10T20:37:28.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIruBmJfRfOBAA5N8KV3c+cLDhcd8GmF2+Jg=="}' 2024-09-10T20:37:28.949 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b1980db4-01ed-4478-bc7d-4914bae05557 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-10T20:37:29.259 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:37:29.276 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-10T20:37:29.277 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=4115e0c7-00a9-4f28-97ce-bb9c8ce7350f --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIruBmJfRfOBAA5N8KV3c+cLDhcd8GmF2+Jg== --osd-uuid b1980db4-01ed-4478-bc7d-4914bae05557 2024-09-10T20:37:29.318 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:29.315+0000 7f43129fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:29.327 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:29.323+0000 7f43129fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:29.328 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:29.323+0000 7f43129fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:29.329 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:29.327+0000 7f43129fd7c0 -1 bdev(0x55d85794c800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:37:29.329 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:29.327+0000 7f43129fd7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-10T20:37:37.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-10T20:37:37.215 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:37:37.216 INFO:tasks.workunit.client.0.smithi039.stdout:adding osd0 key to auth repository 2024-09-10T20:37:37.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:37:37.216 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:37:37.846 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:37:37.847 INFO:tasks.workunit.client.0.smithi039.stdout:start osd.0 2024-09-10T20:37:37.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=9d5dc2fa-7fb6-4340-982d-2ac9a728ee53 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=4f47430c-0c88-433d-949e-db0e81b05862 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=4115e0c7-00a9-4f28-97ce-bb9c8ce7350f --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95628/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:37:37.847 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:37:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:37:37.848 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:37:37.885 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:37.879+0000 7fd27af957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:37.887 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:37.883+0000 7fd27af957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:37.888 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:37.883+0000 7fd27af957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:37:38.337 INFO:tasks.workunit.client.0.smithi039.stdout:0 2024-09-10T20:37:38.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:38.338 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:38.830 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:39.588 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:39.583+0000 7fd27af957c0 -1 Falling back to public interface 2024-09-10T20:37:39.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:39.831 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:39.831 INFO:tasks.workunit.client.0.smithi039.stdout:1 2024-09-10T20:37:39.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:37:39.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:39.832 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:40.322 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:41.323 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:41.324 INFO:tasks.workunit.client.0.smithi039.stdout:2 2024-09-10T20:37:41.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:41.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:37:41.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:41.324 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:41.824 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:42.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:42.826 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:42.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:37:42.827 INFO:tasks.workunit.client.0.smithi039.stdout:3 2024-09-10T20:37:42.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:42.827 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:42.992 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:37:42.987+0000 7fd27af957c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:37:43.304 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:44.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:44.306 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:44.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:37:44.307 INFO:tasks.workunit.client.0.smithi039.stdout:4 2024-09-10T20:37:44.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:44.307 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:45.389 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:46.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:46.391 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:46.392 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:37:46.392 INFO:tasks.workunit.client.0.smithi039.stdout:5 2024-09-10T20:37:46.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:46.393 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:46.863 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:37:47.864 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:37:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:37:47.865 INFO:tasks.workunit.client.0.smithi039.stdout:6 2024-09-10T20:37:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:37:47.865 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:37:47.866 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:37:48.419 INFO:tasks.workunit.client.0.smithi039.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3858260045,v1:127.0.0.1:6803/3858260045] [v2:127.0.0.1:6804/3858260045,v1:127.0.0.1:6805/3858260045] exists,up b1980db4-01ed-4478-bc7d-4914bae05557 2024-09-10T20:37:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:37:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:37:48.420 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:37:48.420 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-10T20:37:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=149228 2024-09-10T20:37:48.421 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-10T20:37:53.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-10T20:37:53.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2024-09-10T20:37:53.990 INFO:tasks.workunit.client.0.smithi039.stderr:pool 'foo' already exists 2024-09-10T20:37:54.007 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:37:55.009 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-10T20:37:55.061 INFO:tasks.workunit.client.0.smithi039.stdout:hints = 1 2024-09-10T20:37:55.061 INFO:tasks.workunit.client.0.smithi039.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-10T20:37:55.061 INFO:tasks.workunit.client.0.smithi039.stdout:Object prefix: benchmark_data_smithi039_150255 2024-09-10T20:37:55.061 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:37:55.061 INFO:tasks.workunit.client.0.smithi039.stdout: 0 0 0 0 0 0 - 0 2024-09-10T20:37:56.061 INFO:tasks.workunit.client.0.smithi039.stdout: 1 16 114 98 0.38277 0.382812 0.158903 0.1436 2024-09-10T20:37:57.061 INFO:tasks.workunit.client.0.smithi039.stdout: 2 16 242 226 0.441344 0.5 0.122704 0.137268 2024-09-10T20:37:58.061 INFO:tasks.workunit.client.0.smithi039.stdout: 3 16 386 370 0.481703 0.5625 0.105571 0.124659 2024-09-10T20:37:59.061 INFO:tasks.workunit.client.0.smithi039.stdout: 4 16 514 498 0.48626 0.5 0.117351 0.125784 2024-09-10T20:38:00.061 INFO:tasks.workunit.client.0.smithi039.stdout: 5 16 624 608 0.474933 0.429688 0.19432 0.130829 2024-09-10T20:38:01.062 INFO:tasks.workunit.client.0.smithi039.stdout: 6 16 738 722 0.469985 0.445312 0.108939 0.131075 2024-09-10T20:38:02.062 INFO:tasks.workunit.client.0.smithi039.stdout: 7 16 896 880 0.491004 0.617188 0.116423 0.126556 2024-09-10T20:38:03.062 INFO:tasks.workunit.client.0.smithi039.stdout: 8 16 1026 1010 0.493094 0.507812 0.117048 0.124897 2024-09-10T20:38:04.062 INFO:tasks.workunit.client.0.smithi039.stdout: 9 16 1152 1136 0.492985 0.492188 0.151391 0.125984 2024-09-10T20:38:05.062 INFO:tasks.workunit.client.0.smithi039.stdout: 10 16 1298 1282 0.50071 0.570312 0.117347 0.124111 2024-09-10T20:38:06.062 INFO:tasks.workunit.client.0.smithi039.stdout: 11 16 1426 1410 0.500638 0.5 0.1426 0.123233 2024-09-10T20:38:07.063 INFO:tasks.workunit.client.0.smithi039.stdout: 12 16 1552 1536 0.499927 0.492188 0.108213 0.124354 2024-09-10T20:38:08.063 INFO:tasks.workunit.client.0.smithi039.stdout: 13 16 1682 1666 0.500529 0.507812 0.142584 0.123846 2024-09-10T20:38:09.063 INFO:tasks.workunit.client.0.smithi039.stdout: 14 16 1808 1792 0.499928 0.492188 0.119274 0.124847 2024-09-10T20:38:10.063 INFO:tasks.workunit.client.0.smithi039.stdout: 15 16 1952 1936 0.504093 0.5625 0.125066 0.123688 2024-09-10T20:38:11.063 INFO:tasks.workunit.client.0.smithi039.stdout: 16 16 2079 2063 0.50359 0.496094 0.114446 0.123974 2024-09-10T20:38:12.063 INFO:tasks.workunit.client.0.smithi039.stdout: 17 16 2210 2194 0.504064 0.511719 0.117441 0.123388 2024-09-10T20:38:13.064 INFO:tasks.workunit.client.0.smithi039.stdout: 18 16 2338 2322 0.503834 0.5 0.117358 0.123511 2024-09-10T20:38:14.064 INFO:tasks.workunit.client.0.smithi039.stdout: 19 16 2482 2466 0.506917 0.5625 0.10929 0.122659 2024-09-10T20:38:15.064 INFO:tasks.workunit.client.0.smithi039.stdout:2024-09-10T20:38:15.062636+0000 min lat: 0.081278 max lat: 0.318493 avg lat: 0.122755 2024-09-10T20:38:15.064 INFO:tasks.workunit.client.0.smithi039.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T20:38:15.064 INFO:tasks.workunit.client.0.smithi039.stdout: 20 16 2610 2594 0.506568 0.5 0.117148 0.122755 2024-09-10T20:38:16.064 INFO:tasks.workunit.client.0.smithi039.stdout: 21 16 2752 2736 0.508854 0.554688 0.153071 0.122635 2024-09-10T20:38:17.064 INFO:tasks.workunit.client.0.smithi039.stdout: 22 16 2880 2864 0.508448 0.5 0.109489 0.122746 2024-09-10T20:38:18.064 INFO:tasks.workunit.client.0.smithi039.stdout: 23 16 3010 2994 0.508418 0.507812 0.167501 0.122406 2024-09-10T20:38:19.064 INFO:tasks.workunit.client.0.smithi039.stdout: 24 16 3138 3122 0.508064 0.5 0.0922509 0.12254 2024-09-10T20:38:20.065 INFO:tasks.workunit.client.0.smithi039.stdout: 25 16 3296 3280 0.512426 0.617188 0.142756 0.121716 2024-09-10T20:38:21.065 INFO:tasks.workunit.client.0.smithi039.stdout: 26 16 3426 3410 0.512245 0.507812 0.0922373 0.121778 2024-09-10T20:38:22.065 INFO:tasks.workunit.client.0.smithi039.stdout: 27 16 3552 3536 0.511499 0.492188 0.153127 0.121671 2024-09-10T20:38:23.065 INFO:tasks.workunit.client.0.smithi039.stdout: 28 16 3682 3666 0.511364 0.507812 0.0920976 0.122051 2024-09-10T20:38:24.065 INFO:tasks.workunit.client.0.smithi039.stdout: 29 16 3826 3810 0.513125 0.5625 0.159171 0.121335 2024-09-10T20:38:25.065 INFO:tasks.workunit.client.0.smithi039.stdout: 30 16 3954 3938 0.512686 0.5 0.125485 0.121562 2024-09-10T20:38:26.065 INFO:tasks.workunit.client.0.smithi039.stdout:Total time run: 30.0714 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Total writes made: 3954 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Write size: 4096 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Object size: 4096 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Bandwidth (MB/sec): 0.513622 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Bandwidth: 0.0486046 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Max bandwidth (MB/sec): 0.617188 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Min bandwidth (MB/sec): 0.382812 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Average IOPS: 131 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev IOPS: 12.4428 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Max IOPS: 158 2024-09-10T20:38:26.066 INFO:tasks.workunit.client.0.smithi039.stdout:Min IOPS: 98 2024-09-10T20:38:26.067 INFO:tasks.workunit.client.0.smithi039.stdout:Average Latency(s): 0.121533 2024-09-10T20:38:26.067 INFO:tasks.workunit.client.0.smithi039.stdout:Stddev Latency(s): 0.0325211 2024-09-10T20:38:26.067 INFO:tasks.workunit.client.0.smithi039.stdout:Max latency(s): 0.318493 2024-09-10T20:38:26.067 INFO:tasks.workunit.client.0.smithi039.stdout:Min latency(s): 0.081278 2024-09-10T20:38:26.163 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-10T20:38:31.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-10T20:38:31.166 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-10T20:38:31.360 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-10T20:38:31.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-10T20:38:31.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-10T20:38:31.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-10T20:38:31.361 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-10T20:38:31.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-10T20:38:31.362 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-10T20:38:31.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4278665216 2024-09-10T20:38:31.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149228 2024-09-10T20:38:31.545 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-10T20:38:32.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149228 2024-09-10T20:38:32.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (149228) - No such process 2024-09-10T20:38:32.546 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-10T20:38:33.049 INFO:tasks.workunit.client.0.smithi039.stderr:osd.0 is already down. 2024-09-10T20:38:33.065 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-10T20:38:33.093 INFO:tasks.workunit.client.0.smithi039.stdout:allocmap bluestore.allocmap 2024-09-10T20:38:35.800 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:38:35.794+0000 7f4111574ac0 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-09-10T20:38:35.801 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:38:35.794+0000 7f4111574ac0 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=7 idx2=8 2024-09-10T20:38:35.801 INFO:tasks.workunit.client.0.smithi039.stderr:2024-09-10T20:38:35.794+0000 7f4111574ac0 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-09-10T20:38:36.449 INFO:tasks.workunit.client.0.smithi039.stderr:allocmap failed: (1) Operation not permitted 2024-09-10T20:38:36.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-09-10T20:38:36.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-09-10T20:38:36.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-10T20:38:36.459 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-bluefs-volume-ops 1 2024-09-10T20:38:36.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-10T20:38:36.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-10T20:38:36.460 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-10T20:38:36.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:38:36.461 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:38:36.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:38:36.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:38:36.462 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:38:36.569 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:38:36.570 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:38:36.570 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:38:36.571 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:38:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:38:36.572 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:38:36.573 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:38:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:38:36.573 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:38:36.574 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:38:36.574 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:38:36.575 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:38:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-10T20:38:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-10T20:38:36.576 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-10T20:38:36.577 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-09-10T20:38:36.578 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-10T20:38:36.581 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:38:36.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:38:36.582 INFO:tasks.workunit.client.0.smithi039.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.95628 2024-09-10T20:38:36.582 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.95628 2024-09-10T20:38:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:36.583 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:38:36.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:38:36.584 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-10T20:38:36.584 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-10T20:38:36.584 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T20:38:36.883 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-10T20:38:37.057 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=63fce0f276bb4bf093b797b7583d8967 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi039 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-09-10T20:38:37.060 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T20:38:37.069 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T20:38:37.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:38:37.131 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T20:38:37.140 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T20:38:37.140 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:38:37.152 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T20:38:37.152 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-10T20:38:37.152 INFO:teuthology.orchestra.run.smithi039.stdout:*hv01.front.sepi 67.205.162.81 3 u 41 128 377 0.078 -1.929 0.879 2024-09-10T20:38:37.152 INFO:teuthology.orchestra.run.smithi039.stdout:+hv02.front.sepi 63.231.80.2 3 u 43 128 377 0.094 -5.157 1.023 2024-09-10T20:38:37.152 INFO:teuthology.orchestra.run.smithi039.stdout:+hv03.front.sepi 74.6.168.72 3 u 102 128 377 0.092 -0.846 1.155 2024-09-10T20:38:37.153 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-10T20:38:37.153 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T20:38:37.162 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T20:38:37.162 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T20:38:37.176 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T20:38:37.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T20:38:37.205 INFO:teuthology.task.internal:Duration was 2870.574892 seconds 2024-09-10T20:38:37.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T20:38:37.218 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T20:38:37.218 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:38:37.254 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T20:38:37.254 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-10T20:38:37.255 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:38:37.306 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T20:38:37.306 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:38:37.397 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T20:38:37.398 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:38:37.452 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T20:38:37.461 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T20:38:37.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:38:37.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T20:38:37.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:38:37.524 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-10T20:38:37.540 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:38:37.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:37.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T20:38:37.594 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T20:38:37.596 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899522/remote/smithi039 2024-09-10T20:38:37.596 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:38:40.491 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T20:38:40.491 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:38:40.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T20:38:40.564 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T20:38:40.564 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T20:38:40.578 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T20:38:40.578 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:38:40.603 INFO:teuthology.orchestra.run.smithi039.stdout: 658640 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 20:38 /home/ubuntu/cephtest 2024-09-10T20:38:40.604 INFO:teuthology.orchestra.run.smithi039.stdout: 788831 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 10 20:00 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:38:40.604 INFO:teuthology.orchestra.run.smithi039.stdout: 790620 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 10 20:26 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T20:38:40.604 INFO:teuthology.orchestra.run.smithi039.stdout: 793457 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Sep 10 20:26 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:38:40.604 INFO:teuthology.orchestra.run.smithi039.stdout: 808419 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Sep 10 20:38 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-10T20:38:40.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:38:40.605 INFO:teuthology.orchestra.run.smithi039.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-10T20:38:40.606 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T20:38:40.611 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T20:38:40.621 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T20:38:40.636 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi039 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T20:38:40.640 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 2870.574892282486 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi039 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=63fce0f276bb4bf093b797b7583d8967 status: fail success: false 2024-09-10T20:38:40.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:38:40.733 INFO:teuthology.run:FAIL